acceptodds
Withdrawn from ICLR 2027

User-Controllable Dense Video Captioning: A Large-Scale Benchmark and Framework

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dense video captioning (DVC) aims to generate temporally localized captions for multiple events in untrimmed videos. Despite recent advances, existing methods produce captions at a fixed granularity because current benchmarks provide only single-style annotations, leaving variations in event density and caption depth largely unexplored. To address this gap, we present User-Controllable Captions (UC Captions), a new dataset with annotations that vary in event density (i.e., how frequently events are localized) and caption depth (i.e., the level of descriptive detail provided for each event). This is the first DVC dataset to explicitly encode controllable dimensions of annotation. Building on this, we propose User-Controllable DVC (UC-DVC), a framework that incorporates user-defined density and depth parameters to dynamically adjust event localization and caption generation. Direct evaluation shows that UC-DVC changes primarily along the requested control axis, produces ordered responses at unseen intermediate inputs, and preserves standard DVC performance across three backbones. These results establish UC Captions as a benchmark for evaluating controllable granularity and UC-DVC as a strong reference framework for the task. To support further research, both UC Captions and UC-DVC code will be publicly released after review.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dense Video Captioning, User-Control, Synthetic Data Generation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_5NaUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends existing video captioning datasets by segmenting videos at varying granularities and controlling caption length, and constructs corresponding embedding modules to enable controllable generation in video captioning models.

Critical strengths

The structure of this paper is quite comprehensive, and the experiments are highly thorough.

Critical weaknesses
  1. The paper emphasizes that density and depth can be provided as continuous inputs; however, in practice, both the dataset annotation and the model training process only utilize three discrete values: 0, 0.5, and 1.
  2. The paper claims that the model can adapt to unseen intermediate inputs. However, in Table 4, one dimension is always fixed at 0.5, and the authors do not evaluate experimental settings where both density and depth are simultaneously unseen (e.g., a (0.25, 0.25) configuration). Furthermore, the choice of 0.25 and 0.75 is rather coarse. While an embedding-based model should inherently possess basic interpolation capabilities, the experiments fail to explore the actual boundaries or limits of this capability.
  3. Current evaluation metrics (e.g., CIDEr, METEOR) are highly sensitive to synonym substitution and length inflation, rendering them incapable of distinguishing between "meaningful detail enrichment" and "meaningless filler text" (or semantic bloating).
  4. if the model predicts "chopping onions" at a control value of 0.25, but outputs "heating oil" and "adding salt" at 0.5 (completely omitting the onion chopping), such "controllability" would be confusing and practically meaningless to the user, even if the total event count increases from 1 to 2. Consequently, the paper currently lacks a quantitative evaluation of the "Semantic Containment Rate" to verify hierarchical consistency across such cross-granularity predictions.
Questions for authors

Table 2 in the paper indicates that MLLM-based DVC methods perform poorly, but this comparison appears to be made against task-specific models under a "fixed granularity" setting. What would be the outcome if a powerful MLLM (e.g., GPT or Gemini) were directly prompted to explicitly generate descriptions at "low/mid/high" density levels, and these outputs were then compared against the generation results of UC-DVC?

Reviewer_fHB3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces UC Captions, a benchmark with nine combinations of event density and caption depth, and UC-DVC, a unified framework for controllable dense video captioning. Experiments cover multiple backbones, controllability, annotation quality, and ablation studies.

Critical strengths
  1. Separating event density from caption depth is a meaningful and practical formulation.
  2. Supporting multiple granularity levels within a single model is useful.
  3. The paper includes evaluations across multiple backbones and several validation experiments.
Critical weaknesses
  1. Inconsistent model attribution in qualitative comparisons. Figure 8 in the ICLR 2026 submission and Figure 9 in this submission show identical displayed predictions, but the baseline changes from HiCM² to Sali4Vid. It is unclear which model produced these results, raising concerns about the reliability of the qualitative comparison.
  2. Unexplained numerical consistency across backbone ablations. Tables 19 and 22 report identical performance improvements across all ablation configurations and metrics for two different backbones, at the reported precision. This is highly unusual for independently trained models and raises serious concerns about experimental reliability.
  3. Unclear human evaluation statistics. Appendix C reports 504 human evaluations across 18 density–depth configurations. However, several reported percentages and mean ratings cannot be reproduced under an evenly distributed design. The per-cell sample sizes and aggregation procedure are not specified, making these results difficult to verify.
  4. Insufficient validation of the proposed benchmark. UC Captions relies heavily on synthetic annotations, but their semantic reliability remains insufficiently established. Event duration, caption length, and CLIP alignment do not adequately verify meaningful event boundaries or factual correctness. Stronger independent validation is needed.
  5. Unclear contribution of architectural design versus model capacity. UC-DVC introduces approximately 35.5M additional parameters through its density and depth encoders. However, it remains unclear whether the performance gains come from the proposed architecture or simply increased model capacity. Comparisons with parameter-matched baselines using simpler control mechanisms are needed to isolate the architectural contribution.
Questions for authors
  1. Which backbone produced the predictions in the reused qualitative figure?
  2. What explains the identical ablation improvements across different backbones? Can the original evaluation records be provided?
  3. How were the human evaluation scores calculated, and how many judgments were collected per configuration?
  4. How is the semantic reliability of the synthetic annotations independently verified?
  5. Can simpler control-conditioned architectures achieve comparable results under the same training conditions?
Reviewer_aT4wOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces UC Captions, a dense video captioning (DVC) benchmark derived from YouCook2 and ViTT that annotates each retained video on a 3×3 grid of event density and caption depth (low/mid/high), yielding substantially more event-caption pairs than the source datasets. It further proposes UC-DVC, a framework that injects continuous density and depth control embeddings into existing DVC backbones (Vid2Seq, HiCM², Sali4Vid) together with density-aware sequence loss and depth-aware comparative loss, and reports axis-isolated controllability, preservation of standard DVC metrics, and ablations.

Critical strengths
  1. Explicitly separating event density (how many events are localized) from caption depth (how detailed each caption is) is a practical and intuitive formulation that matches real user variation.

  2. Training a single model that can respond to different requested granularities is more convenient than training nine separate models.

Critical weaknesses

W1 The task itself feels under-defined and too engineering-oriented. Basically it just controls how many events you get and how long each caption is. There is almost no discussion of hierarchical meaning or whether finer events are real refinements of coarser ones. This kind of formulation doesn’t generalize much beyond the specific dataset they built.

W2 The method is pretty incremental. They mainly stick two continuous embeddings into existing models and add a density-weighted loss plus a depth ranking loss. Most of the gain seems to come from the multi-style training data rather than the architecture itself. Without the control interface the performance drops a lot, which shows the model is mostly riding on the new annotations.

W3 Validation of the synthetic captions is still not strong enough. A lot of the 3×3 grid is generated by MLLM+LLM. Length, duration and CLIP scores don’t really prove the event boundaries or the factual details are reliable. The human study is also a bit hard to interpret from the numbers given.

Questions for authors

Can you show that higher-density predictions actually contain the semantics of the lower-density ones, not just more segments?

Have you tried simpler control methods (like prompt tokens) with similar parameter count? Do they get close results?

Reviewer_2FFtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces two controls for dense video captioning: event density and caption depth. UC Captions expands YouCook2 and ViTT into a 3×3 annotation grid through temporal merging/splitting and an MLLM–LLM caption-generation pipeline. UC-DVC adds separate conditioning pathways and granularity-aware objectives to existing DVC backbones. The evaluation measures control adherence, behavior at unseen intermediate inputs, performance across the annotation grid, and standard DVC performance. The authors also assess the constructed annotations using automatic checks and human judgments.

Critical strengths
  • Separating event density from descriptive detail is a useful task formulation. Table 4 tests the two controls independently: on YouCook2, density changes the event count from 5.24 to 9.94 with little change in caption length, while depth changes words per event from 3.12 to 16.56 at a fixed event count. The unseen-input experiments and per-video correlations provide additional evidence that the model responds to the controls.

  • The paper puts considerable effort into validating the annotations and testing the framework across backbones. The human quality assessment and anchor-blinded study are useful additions to the automatic metrics. Standard DVC performance also improves: with Sali4Vid on YouCook2, CIDEr/F1 rises from 76.13/33.79 to 82.35/36.91.

Critical weaknesses
  • The factual quality of added detail needs a more direct assessment. The generation prompt gives the depth levels markedly different word budgets, so length provides a strong cue for level identification. Human quality ratings help, but the study does not report how often newly added details are visually correct. The NLI check compares against an incomplete source caption, CLIP measures broad alignment, and the Gemini depth prompts do not include the video. At the model-output level, Figure 12 shows why this matters: the named tool changes from a food processor to a pestle and mortar and then back as depth increases. This example does not establish a general error rate, but it shows that changing depth can change facts rather than simply add information. For a benchmark centered on descriptive detail, that distinction needs to be measured.

  • A straightforward conditioned baseline is missing. Training on all cells without control inputs gives the model several annotation styles that it cannot distinguish. Training separate models on individual cells removes the opportunity to share supervision across the grid. Neither comparison answers how well ordinary density/depth control tokens would work with the same Full UC data. Table 19 usefully ablates the losses, but retains the proposed architecture. Given the additional 35.5M parameters, the simpler conditioned baseline is needed to establish what the dedicated pathways contribute.

Questions for authors
  1. Can you report a focused, video-grounded audit that separates correct added details, unsupported additions, and changes to existing facts across depth levels? Please distinguish constructed annotations from model outputs, explain the tool change in Figure 12, and clarify how many independent videos underlie the human judgments.

  2. How does the original backbone perform with density/depth control tokens and ordinary cross-entropy when trained on the same Full UC annotations for the same number of updates? Both control adherence and caption/localization quality would be informative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.