acceptodds
Withdrawn from ICLR 2027

Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Linking people’s appearance and actions to their character identities is essential for understanding video narratives. We present a framework for identity-aware video captioning and person-centric question answering that combines automatic character identification, explicit spatial grounding, and task-specific adaptation of video-language models. Starting from movie clips in LSMDC v2, our pipeline matches detected faces to actor reference images, tracks characters across frames, and constructs shot-level inputs with identity-linked bounding boxes. We use a strong vision-language model to generate identity-aware captions and associated questions, followed by manual verification and filtering to obtain an evaluation benchmark of 750 captioned clips and 3,000 person-centric questions. We investigate five grounding strategies combining textual coordinates with visual face or estimated person boxes, and evaluate them across multiple Video-MLLM families at approximately 2B, 4B, and 8B parameters, as well as larger frontier models. Combining visual face boxes with textual coordinates provides the most consistent performance across scales and significantly improves question-answering accuracy over coordinates alone. We further observe that smaller models tend to over-assign known identities when the queried person is not grounded, whereas larger models better distinguish such UNIDENTIFIED cases. Building on the strongest grounding formulation, we introduce BAC by LoRA fine-tuning Qwen models at 2B, 4B, and 8B scales on approximately 32K identity-aware captioned clips. BAC-8B reaches 93.20% overall QA accuracy, ranking behind only GPT-5.6 Sol among the evaluated frontier models while outperforming the other open-weight and proprietary systems. Overall, our results show that explicitly communicating who is where, together with lightweight task-specific adaptation, substantially improves identity-aware video understanding without modifying the underlying model architecture. We publicly release the training data, evaluation benchmark, source code, and trained BAC checkpoints.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-Language Models, Identity-Aware Video Captioning, Visual Grounding, Person-Centric Question Answering, Multimodal Learning, Character Identification, Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uruZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a pipeline for identity-aware video captioning and person-centric question answering, where known character identities and locations are supplied to a Video-MLLM through explicit spatial cues. The authors construct an LSMDC-derived benchmark of 750 manually verified shots and 3,000 questions, compare five identity-grounding representations within the Qwen3-VL family at three model scales, and introduce BAC through caption-only LoRA fine-tuning. Cross-family and frontier-model comparisons use face visual boxes combined with textual coordinates. Evaluation covers captioning and identity-targeted QA. The reported results show benefits from visual grounding over coordinates alone and further improvements from task-specific adaptation on the constructed benchmark.

Critical strengths
  1. A practically relevant task formulation and evaluation resource. Associating descriptions and actions with the correct characters is important for useful movie understanding. The benchmark evaluates character attribution directly and includes people who cannot be assigned a supplied identity. Manual verification strengthens its usefulness as an evaluation resource. This is a substantive contribution because correct character attribution can be obscured by conventional caption-quality metrics.
  2. Meaningful benefits from lightweight adaptation, including transfer between tasks. BAC is trained on captions but improves person-centric QA across three model sizes. This provides evidence that the benefit extends beyond reproducing the directly supervised output task. The matched-backbone comparisons support the practical value of adaptation for compact models, even though the mechanism and broader transfer of the improvement remain insufficiently characterized.
Critical weaknesses
  1. The evaluation does not sufficiently distinguish temporal identity association from per-frame recognition and adaptation to the annotation protocol. Identity-targeted QA measures attribution accuracy, but higher aggregate accuracy does not establish that the model must associate identities across time. The benchmark consists of short, single-shot videos, and questions about appearance, posture, position, and some actions may be answerable from one informative frame. Short duration alone does not establish simplicity; however, without a single-frame comparison, the contribution of temporal information remains unclear. The category-wise analysis in Appendix D does not resolve this issue, since an “action” or “interaction” question is not necessarily temporally dependent. More diagnostic evidence would evaluate questions requiring identity information from a different frame, particularly when the queried person lacks an annotation in the frame containing the relevant action. Caption-to-QA transfer is encouraging, but both tasks share the grounding convention, so this transfer does not rule out adaptation to that convention as an explanation for some gains. A matched identity-agnostic caption adaptation control, alternative input representations, and consistently remapped identity labels would help establish what capability BAC acquires beyond more reliable use of supplied spatial cues.
  2. The benchmark informs method selection and is subsequently reused for final evaluation, leaving the selected recipe without independent confirmation. The same benchmark is used to compare grounding representations and identify the strongest backbone. The selected representation and backbone then underpin BAC, which is fine-tuned on separate training data and evaluated on that benchmark. Consequently, the final evaluation is not independent of the decisions used to construct the method. Selecting configurations according to their performance on a finite evaluation sample can favor configurations that benefit from sample-specific characteristics; evaluating the resulting method on the same sample cannot establish whether that advantage generalizes. This concern is distinct from training directly on test examples. The separate validation split used for LoRA hyperparameters and checkpoint selection does not address the earlier grounding and backbone decisions. The reported within-benchmark improvements remain meaningful, but they provide weaker evidence for the general superiority of the selected recipe. Selecting these components on development data, freezing the choices, and evaluating on an untouched test set would provide stronger confirmation. The cross-group comparison compounds this limitation. The five representations are systematically compared only within Qwen3-VL, while other groups receive the selected FVB+FTC configuration. Applying the same configuration to every model does not establish that it is equally suitable for each group. Thus, the rankings are conditional on a representation selected based on Qwen, and the abstract suggests broader validation of the grounding strategies than the experiments provide. Independent data or annotation-protocol validation is therefore particularly important. Suitable adaptations should preserve comparable identity and spatial information, and dataset overlap should be checked explicitly. This also matters for novelty: visual region labels and learning to interpret them have precedents [1], [2]. Combining established components can be valuable, but the paper needs stronger evidence of what new capability or broadly applicable empirical insight its known-identity formulation contributes.
  3. The presentation does not clearly delineate the framework and the responsibilities of its components. The constituent methods are described, but the overall pipeline and its boundaries are difficult to follow. Figure 1 illustrates a task example rather than the complete framework. An integrated diagram should distinguish offline character identification and tracking, synthetic annotation and benchmark construction, BAC adaptation, and inference-time inputs and outputs. It should make explicit which identity correspondences are supplied by preprocessing and which associations BAC must infer. Clear input specifications and promptexamples would also help readers understand the role of FVB+FTC during training and evaluation. This is a presentation concern but it affects the assessment of the model’s claimed contribution.
Questions for authors
  1. Clarification: What proportion of questions genuinely requires information from multiple frames? How are identity answers verified when the queried person is unannotated in the relevant frame but has an identity-linked annotation elsewhere in the shot?
  2. Additional evidence for the current capability claim: How do BAC and its base model compare under full-video and single-frame inputs, particularly on questions requiring cross-frame identity association?
  3. Additional evidence concerning the source of improvement: Does BAC retain its advantage under alternative grounding representations or consistently remapped identity labels? Can a matched identity-agnostic caption adaptation control help separate identity-linked learning from general task and format adaptation?
  4. Additional evidence concerning generality: Does the grounding-strategy finding hold in another model family, or does BAC retain its gains on an independent evaluation protocol with suitable input adaptation and verified absence of training-data overlap? Either result would materially strengthen the scope of the contribution.
Reviewer_61WhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies identity-aware video captioning and person-centric video QA, where character names must be connected to the correct people and actions in movie shots. It builds an LSMDC-derived benchmark with identity-linked spatial annotations, compares five textual and visual grounding representations, and introduces BAC, LoRA-adapted Qwen3 models. The evaluation measures captioning and QA performance across model scales and compares against frontier systems.

Critical strengths

This work systematically evaluates five spatial grounding strategies across three Qwen3-VL model sizes, providing useful empirical insights into how visual and textual grounding cues affect identity-aware video understanding. It also shows that LoRA fine-tuning on identity-aware captions improves both captioning and QA across three Qwen3 model scales, despite the absence of QA-specific supervision.

Critical weaknesses
  1. The proposed pipeline primarily combines existing face recognition and tracking techniques, visual bounding-box annotations, and standard LoRA fine-tuning, without introducing new methodology. Additionally, it remains unclear how much of the improvement comes from identity-aware supervision rather than conventional task-specific fine-tuning.

  2. The reliability of the automatically constructed identity annotations is questionable. Identities are assigned by matching tracked InsightFace embeddings to web-collected actor reference galleries. The paper does not report necessary details of this procedure, e.g., identity switches, missed characters, or whether manual review corrected the problematic mappings and boxes. This is important as a mistaken label can turn a correct visual association into an incorrect QA target, or vice versa.

Questions for authors
  1. Could the authors clarify whether the Pi-to-name mapping is explicitly included in the prompt during captioning and QA inference, for each grounding representation as well as for the BAC training inputs and targets? If names are introduced only through post-hoc replacement, what does the face-to-actor name matching contribute beyond consistent track labels?

  2. Some questions appear answerable from the gender of the candidate names alone, e.g. “Who is pregnant?” in Fig. 3 has a single female candidate among four. What is the QA accuracy on the subset where such cues do not disambiguate? This would verify that the claimed gains reflect identity grounding rather than name-based shortcuts.

  3. Gemini 2.5 Flash achieves 82.63% overall QA accuracy, below the untuned Qwen3 8B (87.70%), yet BAC-tuning on its generated captions still improves the 8B model to 93.20%. What motivated the choice of Gemini as the caption generator, and what do you believe the 8B model learns from a teacher that is weaker on the downstream task?

Reviewer_h7fqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies identity-aware video captioning and person-centric question answering. The authors construct a pipeline that identifies movie characters through face recognition and tracking, and then provides identity-linked spatial annotations to Video-MLLMs. Five grounding representations are compared across different model scales and families. The authors also introduce BAC, which fine-tunes Qwen3-VL models using approximately 32K generated identity-aware captions. Evaluation is conducted on a manually curated benchmark of 750 video clips and 3,000 questions. BAC-8B achieves 93.20% QA accuracy, outperforming several larger models on the proposed benchmark.

Critical strengths
  1. Useful benchmark and detailed evaluation.

The paper provides a curated benchmark covering multiple identity-related question types, including UNIDENTIFIED cases. The comparisons across grounding strategies and model scales offer useful insights into how Video-MLLMs utilize explicit identity information.

  1. Consistent empirical improvements.

BAC achieves substantial improvements over the corresponding Qwen3-VL baselines, despite being trained only on captioning data. The improvements in UNIDENTIFIED accuracy are particularly interesting, suggesting some transfer from caption supervision to identity-related QA.

Critical weaknesses
  1. Limited methodological novelty and unclear contribution beyond explicit identity annotations.

The central approach relies on existing face recognition and tracking methods to establish character identities, followed by bounding-box visualization, coordinate prompting, and standard LoRA fine-tuning. Much of the identity association is therefore provided to the model rather than learned by it. While this is a reasonable engineering approach, I find the methodological contribution relatively incremental. The paper also lacks experiments examining how errors in the upstream identity annotations affect downstream performance.

  1. Insufficient controls to establish the source and generality of the improvements.

BAC is fine-tuned on approximately 32K identity-aware captions, whereas the compared baseline models receive no corresponding task-specific adaptation. The reported improvements are meaningful, but it remains difficult to separate the effects of supervision, grounding representation, and their combination. In particular, matched fine-tuning ablations using alternative grounding strategies are missing. Moreover, captioning quality relies entirely on a VLM judge without reported human agreement, which limits confidence in the captioning conclusions.

Questions for authors
  1. How accurate are the automatically generated identity annotations, and how does BAC perform when some identities or bounding boxes are incorrect or missing?

  2. Can the authors provide matched LoRA fine-tuning results using FTC-only or FVB-only grounding to isolate the contribution of the selected FVB+FTC representation?

  3. Has the VLM-based caption evaluation been validated against human judgments? How sensitive are the reported results to the choice of judge?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.