acceptodds
Withdrawn from ICLR 2027

CRTrack-Cap: A New Benchmark for Cross-View Dense Video Object Captioning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dense video object captioning (DVOC) localizes, tracks, and describes object trajectories within a video. Existing methods process each camera view independently, producing separate camera-local tracklets and captions for the same person across cameras. These outputs lack a unified description and leave complementary appearance evidence across views unused. We therefore formulate cross-view DVOC (CVDVOC), which associates these tracklets into scene-level global trajectories and generates one caption for each trajectory from synchronized views. To support this task, we construct CRTrack-Cap from CRTrack by auditing text-identity links and converting usable referring expressions into canonical captions. It retains 425 global trajectories and 838 local tracklets across 13 scenes, with canonical captions for 335 identities. We adapt CHOTA to the cross-view setting as CV-CHOTA to jointly evaluate tracking and caption quality. Our reference framework combines View-Exclusive Cross-Camera Association (VECA) for grouping tracklets with Conflict-Evidence Gating (CEG) for integrating caption evidence using its reliability and disagreement across views. Against three single-view methods adapted with MvMHAT for cross-view association, our framework achieves CV-CHOTA scores of 83.43 and 33.48 on the scene-disjoint in-domain and cross-domain tests, respectively. Together, CRTrack-Cap and our framework provide a benchmark and a strong baseline for describing the same person consistently across camera views.

Primary area:
datasets and benchmarks
Keywords:
Dense Video Object Captioning, Dataset and Benchmark, Cross-View Multi-Object Tracking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9VDSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces cross-view dense video object captioning, which associates observations of the same target across cameras into a global trajectory and generates one caption for it. The authors construct CRTrack-Cap from CRTrack, curating reference captions for 335 of its 425 global trajectories, and extend CHOTA to cross-view evaluation. The reference framework combines VECA and CEG and achieves the highest CV-CHOTA among the compared methods on both the in-domain and cross-domain tests.

Critical strengths
  • The task is well motivated. Different cameras can provide complementary appearance information, and combining cross-camera identity association with object captioning is a useful extension of single-view DVOC.
  • The dataset retains links between captions and the original identity annotations, specifies which identities are eligible for caption evaluation, and provides scene-disjoint in-domain and cross-domain splits. These choices provide a useful basis for future work.
Critical weaknesses
  • Some experimental details are not fully described. Section 5.1 discusses tracker fine-tuning and association with MvMHAT, but it is unclear how each baseline produces a single global caption from its local outputs. The training settings for the captioning modules are also insufficiently specified. These details affect caption scores and are needed to reproduce the results and interpret the performance gaps in Table 2.
Questions for authors
  • How does each baseline convert its local outputs from multiple views into one global caption? Are the captioning modules fine-tuned on CRTrack-Cap, and what training data and settings are used?
  • Under Eq. (4), the cross-domain VECA results in Table 3 imply that the combined METEOR and SPICE improvement of CEG over MVMean is approximately 4.94. In Table 6, the corresponding improvements are 3.40 and 3.11 for the three and four-view groups. Do these groups cover all 49 cross-domain identities, and were they evaluated using the same predictions, matching, and per-identity scoring as Table 3? Please report the group sizes and explain how these values can be reconciled if these conditions hold.
  • Please expand the related work discussion to cover cross-view caption consistency [1] and object captioning from multiple views [2, Fig. 6(b)], and clarify how these works relate to and differ from the proposed task.

[1] Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions. ICCV 2025.

[2] Describe Anything: Detailed Localized Image and Video Captioning. ICCV 2025.

Reviewer_CWfMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces cross-view dense video object captioning, which associates person tracklets across synchronized camera views and generates one appearance caption per global trajectory. The authors construct CRTrack-Cap with 425 global trajectories, including 335 canonical captions, and propose CV-CHOTA for joint tracking and caption evaluation. They also introduce a reference framework combining VECA for cross-view association and CEG for caption fusion, achieving the highest CV-CHOTA among the compared methods on both in-domain and cross-domain tests.

Critical strengths

A clearly defined task with careful data auditing. The paper presents a well-motivated cross-view extension of dense video object captioning. Its explicit handling of ambiguous, shared, and missing text–identity mappings makes the benchmark construction transparent and the evaluation population clearly defined.

Critical weaknesses
  • Insufficient external validation. All experiments are conducted on the authors’ own CRTrack-Cap benchmark, using the proposed task formulation and evaluation protocol. It therefore remains unclear whether the claimed benefits of cross-view association and caption fusion generalize beyond this setting. The paper should provide at least one form of external validation, such as evaluation on an independent multi-camera dataset, human assessment of cross-view identity and caption correctness, or a downstream video-understanding task that directly benefits from the generated global trajectories and captions.
Questions for authors
  1. Given that only 196 captioned identities are available for training, is it necessary to fine-tune approximately 107.61M parameters for 70 epochs? Have the authors analyzed whether a much smaller adaptation module would suffice, as well as the effects of model size on overfitting and generalization?

  2. The benchmark appears to reuse CRTrack’s videos and annotations, with canonical captions derived from existing referring expressions. Could the authors clarify what new data or annotations were actually created, and how their validity and quality were verified? If the primary contribution is the pipeline, what specific advantages does it demonstrate over simpler alternatives, and how efficient is it in terms of runtime and computational cost compared with the baselines?

Reviewer_iuFxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

the paper introduces cross-view dense video object captioning. existing systems caption each camera separately, so the same person gets one track and one caption per view, and details visible in only one camera get thrown away. the authors pose the task as producing one caption per person across all synchronized cameras and build a benchmark for it, crtrack-cap, on top of crtrack. rather than assigning all referring expressions to people, they audit the text-identity link in both directions, turn usable ones into canonical appearance captions, and leave identities without usable text with a null caption instead of guessing. the benchmark spans thirteen scenes with scene-disjoint in-domain and cross-domain test splits. their system tracks people per camera, uses veca to link local tracks of the same person across cameras while keeping at most one track per camera in a group, and ceg to weight caption evidence by reliability and cross-view agreement. they also adapt chota into cv-chota to score detection, association and caption jointly. the results beat the best adapted baseline on both test splits, and the component study on fixed local tracklets shows ceg lifting caption quality over plain averaging on both splits.

Critical strengths

first, the task is genuinely new and well motivated. dvoc is an active area, the observation about discarded complementary evidence is correct and useful, and having a benchmark plus a baseline for the cross-view version is a contribution others can build on. this feels like a paper that opens a direction rather than one that squeezes out a couple of points.

second, the data work is more careful than most benchmark papers. the audit runs both directions, shared expressions keep their group scope instead of being handed to one person, identities without usable text keep their tracking annotations and get null captions, and the appendix even records source defects such as clipped boxes and missing frames. the component study on frozen local tracklets is the right way to separate association from caption fusion, and reporting a case where their method is not best makes the rest more believable.

Critical weaknesses

the main table is not controlled: the baselines use mvmhat while the proposed system uses veca, and local tracker and caption pathway differ too, so the headline gain changes three things at once. table 3 and appendix table 5 do compare association rules, but veca's incremental contribution over an established method such as mvmhat under otherwise controlled conditions remains unclear. the appendix numbers also suggest most of the gain comes from a simple view-exclusive rule: greedy matching already reaches most of the way to full veca. that does not isolate the one-track-per-camera constraint, since greedy also assigns across cameras, but it makes the elaborate merge rule look less essential. similarly, ceg is only ever compared against plain averaging as a whole, so a simpler version might do nearly as well.

second, the benchmark layer is unvalidated and small. no systematic human validation of the canonical captions is reported, yet all caption metrics and the joint score are computed against them, so errors may flow into the conclusions. the data is also modest: thirteen scenes, a single validation scene, and only forty-nine captioned identities in the cross-domain test, where the margin over the best baseline is about two and a half points and no error bars are given. the cross-domain split changes source collection and camera setup together, so domain shift and tracking degradation are confounded, and the tracking drop is an observed outcome rather than an independently varied condition. finally, a single aggregate score can hide where the differences lie: the proposed method reaches a higher cross-domain cv-chota than smoter even though smoter tracks better, and global-capa mixes metrics on different scales.

Questions for authors

could you run your own pipeline with mvmhat instead of veca on identical local tracklets, and how much of the gain comes from a plain view-exclusive rule versus the full merge procedure? do the individual pieces of ceg matter, especially the reliability term and the disagreement term? is there any systematic human validation of the canonical captions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.