acceptodds
Under review as a conference paper at ICLR 2027

A Multi-Domain Adaptation Benchmark for Video Grounding in VLMs

Abstract

Vision-Language Models (VLMs) have shown strong performance in Spatio-Temporal Video Grounding (STVG), yet they are still evaluated mostly in a zero-shot manner on general-purpose benchmarks of everyday scenes. This creates a critical disconnect from real-world applications in specialized domains, where models inevitably encounter rare visual or textual concepts. Since exhaustive pre-training across infinite data distributions is infeasible, the ability to adapt to novel domains with limited data is essential. To bridge this gap, we introduce AnyGroundBench, a domain-adaptation benchmark designed to shift the STVG evaluation paradigm from static zero-shot testing to rigorous domain adaptation. Targeting five specialized domains (animal, industry, sports, surgery, and public security), AnyGroundBench pairs newly captured, expert-annotated videos with established datasets, unifying them through dense, high-fidelity spatio-temporal annotations. Crucially, the benchmark provides dedicated limited training subsets, enabling systematic evaluation of domain adaptability under limited training data. We benchmark 23 state-of-the-art VLMs in the zero-shot setting and further evaluate five adaptation strategies, spanning training-free and fine-tuning-based approaches, on representative models, assessing their zero-shot generalization and adaptation capacity. Our results show that current VLMs remain far from practical performance in the zero-shot setting, that training-free adaptation produces highly variable effects depending on the model and domain, and that fine-tuning-based adaptation, though more effective, still falls short of real-world requirements, with gains varying markedly across domains. These findings expose fundamental limitations in current VLMs' spatio-temporal reasoning, pointing to concrete directions for future research. The code and dataset are available at https://github.com/anonymous7bhblm-cpu/7bnblm and https://huggingface.co/datasets/anonymous7bhblm/7bnblm.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Spatio-Temporal Video Grounding, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WZQVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AnyGroundBench evaluates video grounding and adaptation across five specialized domains, with 2,042 videos, 3,525 queries, and 23 VLMs. It finds that precise spatio-temporal grounding remains difficult after adaptation.

Critical strengths
  1. Specialized-domain grounding with limited training data is a useful benchmark setting.
  2. The supplement adds meaningful evidence: split descriptions, annotation QC agreement, parsing statistics, and ICL ablations. Code and data links are available.
Critical weaknesses
  1. Split independence and model selection remain unclear. Appendix C specifies video-level splits, but not whether clips from the same recording session, mouse, or game can cross splits. GEPA’s validation data and stopping rule also need clarification. These details affect what the adaptation results establish.
  2. The claim of a fundamental joint-reasoning limitation is stronger than the controls support. SVG receives ground-truth temporal crops, while STVG sees full clips; model frame budgets also differ. The parsing audit helps, but does not isolate these effects. Post-rebuttal update (19 Nov 2026): The authors’ clarifications and additional controls address my main concerns, and I now recommend weak acceptance.
Questions for authors
  1. What source/session grouping defines train/test independence, and which held-out data are used for prompt or checkpoint selection?
  2. Does the SVG/STVG gap persist with matched frame budgets and oracle temporal localization, or a composed localization-and-tracking baseline?
Reviewer_k6QYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AnyGroundBench, a benchmark for evaluating spatio-temporal video grounding (STVG) and adaptation in five specialized domains: animal, industry, sports, surgery, and public security. The benchmark contains 2,042 videos and 3,525 queries, combining eight existing datasets with two newly collected datasets. It provides domain-specific training and test subsets and evaluates three related tasks: STVG, temporal video grounding (TVG), and spatial video grounding (SVG). The authors benchmark 23 VLMs and evaluate five adaptation strategies, including in-context learning, prompt optimization, supervised fine-tuning, and reinforcement fine-tuning. The benchmark addresses a relevant evaluation gap, and the experiments reveal substantial performance limitations in specialized-domain grounding. However, the evidence supporting the domain-adaptation protocol and the interpretation of the identified bottlenecks requires further clarification. My main concerns involve split integrity, the limited-data evaluation design, inference and metric standardization, and the strength of the conclusions drawn from separately evaluated subtasks.

Critical strengths

Strengths 1.Relevant benchmark motivation. Evaluating whether VLMs can adapt to specialized grounding tasks is practically meaningful. Providing target-domain training subsets makes the benchmark useful for studying adaptation beyond static zero-shot performance.

2.Complementary domain coverage. The five domains introduce different challenges, including small surgical instruments, rapid animal movements, industrial terminology, and crowded sports scenes. Combining existing resources with newly collected mouse and American football videos adds useful diversity.

3.Substantial annotation effort. The paper describes a detection-and-tracking pipeline followed by manual correction and additional verification. Involving medical experts and experienced football players for the newly collected subsets is particularly appropriate for actions that require specialized knowledge.

4.Broad empirical coverage and useful diagnostic tasks. Evaluating 23 VLMs and several adaptation approaches provides a useful initial reference. Reporting TVG and SVG alongside STVG also helps reveal differences that an aggregate STVG score would conceal.

Critical weaknesses

Weaknesses 1.The adaptation protocol and split construction are insufficiently specified in the available manuscript. Section 3.2 defines adaptation broadly, but the main text does not explain whether training and test data are separated by original recording, subject, match, surgical procedure, or recording session. Since multiple clips and queries can originate from the same recording, clip-level separation alone may allow substantial overlap in visual context. This is especially important for a benchmark whose central contribution is measuring adaptability. The intended setting should also be stated more precisely: supervised adaptation to new domains, generalization to unseen concepts within a domain, and generalization across recording environments are different evaluation targets.

2.The limited-data adaptation claim is not supported by a systematic data-efficiency study. Table 2 provides approximately 410–583 training queries per domain, but Table 4 reports only one training-set size. These results cannot establish how adaptation scales with the available supervision or whether the observed limitations persist across data budgets. Furthermore, dense tube annotations provide substantially different amounts of supervision depending on event duration. Fine-tuning is evaluated only on Qwen3-VL-8B, which also limits the generality of the conclusion that adaptation produces only modest STVG gains. A benchmark paper need not introduce a new adaptation method, but stronger characterization of its adaptation setting is important here.

3.The claimed shift toward a joint spatio-temporal alignment bottleneck is not established by the current decomposition. SVG receives a video cropped to the ground-truth temporal interval, whereas STVG receives the full video. SVG therefore benefits from oracle temporal localization, potentially different effective sampling density, and less distracting context. In addition, the three tasks use separate prompts and predictions. Consequently, strong standalone SVG and TVG results combined with weak STVG do not, by themselves, isolate a joint-alignment failure. Joint metrics also penalize simultaneous spatial and temporal inaccuracies. Controlled oracle experiments using the same STVG predictions would provide stronger evidence for this interpretation.

4.Important evaluation choices and baseline coverage remain unclear. The paper explicitly notes that some models predict boxes only at selected timestamps, but the available text does not specify how these predictions are converted into dense tubes. Interpolation, missing-box handling, coordinate conversion, output parsing, and truncation can materially affect the results. Frame sampling and resolution are also particularly consequential for rapid actions and small objects. Moreover, Table 3 includes only one specialized STVG VLM, LLaVA-ST, despite discussing several grounding-oriented models. Additional specialized baselines and a modular temporal-localization-plus-detection/tracking pipeline would help determine how much of the observed difficulty comes from domain shift versus the chosen direct-generation interface.

5.Annotation quality and domain-specific difficulty need stronger quantitative validation. The described manual review process is encouraging, but the available manuscript doess not report quantitative agreement for temporal boundaries, target identity, or bounding boxes. Queries adapted from action labels may also be ambiguous when several similar objects or actors are present. Additionally, Figure 4 shows that event duration and object size affect performance. Without source-level results or controlled comparisons, differences attributed to specialized-domain understanding may partly reflect object size, motion, viewpoint, or temporal scale.

6.Uncertainty is not reported for the central adaptation comparisons. Each domain has only 157–250 test queries, so some reported differences correspond to a small number of successful examples. Although Figure 4 includes bootstrap confidence intervals, Tables 3 and 4 do not report uncertainty, and repeated adaptation runs are not described in the supplied text. This makes small improvements and degradations difficult to interpret. If several queries share a video or recording, uncertainty estimation should also account for this dependence rather than treating all queries as independent.

Questions for authors

Questions that need the author to answer

1.How exactly are the train, validation, and test splits constructed? Are they disjoint at the original-recording level? What additional separation is enforced for animals, participants, football matches, surgical procedures, and recording environments? Are official source-dataset splits preserved?

2.What is the precise adaptation setting and supervision budget? Are separate models trained for each domain and each task, or is a shared model used? How are checkpoints, prompts, and hyperparameters selected without test-set feedback? Can the authors provide learning curves across several training-set sizes?

3.Can the authors directly test the joint-alignment hypothesis? Please report results after replacing the temporal boundaries of STVG predictions with ground truth, and after replacing their spatial boxes with ground truth under an explicitly defined evaluation protocol. How does a sequential TVG-to-SVG pipeline compare with direct STVG prediction?

4.How are inference inputs and sparse outputs standardized across models? Please specify frame sampling, resolution, timestamp representation, output-length limits, interpolation, and treatment of invalid or incomplete predictions. Would the main findings remain consistent under alternative sampling budgets and stricter IoU thresholds?

5.What quantitative evidence supports annotation reliability? Please provide agreement measurements for temporal spans, bounding boxes, and target selection, together with rules for repeated events, occlusion, multiple matching instances, and ambiguous queries. A small human-performance study would also help contextualize benchmark difficulty.

6.How robust are the adaptation results to experimental variation and baseline choice? Can the authors report paired confidence intervals, multiple fine-tuning seeds, and variability from demonstration selection or prompt optimization? Would the conclusions hold for another adaptable backbone, an additional specialized grounding model, and a modular grounding pipeline?

Reviewer_CERuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose AnyGroundBench, a benchmark for evaluating spatio-temporal video grounding (STVG) in VLMs under domain adaptation. Their key motivation is that VLMs are evaluated mostly zero-shot on everyday scenes, whereas real deployments require adapting to specialized domains with limited data. The authors decompose STVG into spatial and temporal sub-tasks, evaluate 23 VLMs zero-shot, and evaluate five existing adaptation strategies on their benchmark. They find that zero-shot STVG remains far from practical, that spatial localization is the main bottleneck for most models.

Critical strengths

The proposed benchmark is timely and addresses an important gap. As performance on general web video benchmarks saturates, evaluating VLMs in specialized domains is increasingly necessary

Evaluating on the difficult STVG task is appreciated, as it moves beyond the multiple-choice format used by many existing VLM benchmarks

The newly collected and expert-annotated data is a useful contribution

The evaluation is broad, covering 23 VLMs and both training-free and fine-tuning-based adaptation strategies

Critical weaknesses

W1: The paper attributes low performance of zero-shot open-source VLMs to the specialized nature of the domains. However, to the reviewer, a reasonable yet unexplored explanation is that these models were not trained for the tasks being evaluated. The authors show in Table 4 that fine-tuning on domain-specific STVG data helps quite a bit (especially on tasks where the zero-shot model is at 0%), but the reviewer thinks fine-tuning on a similar scale of general STVG data could perform similarly. This could be explored by fine-tuning on HC-STVG videos (or any STVG training dataset that isnt aligned with the domains in AnyGroundBench), and testing on AnyGroundBench

W2: For the self-collected data, the reviewer is concerned that they might be testing adaptation to a recording setup rather than a domain. The newly captured datasets are being split randomly at the video level, so the training and test sets likely share the same animals, cages, cameras, or games. Is it possible that the gains on these domains may therefore reflect fitting a specific recording setup rather than adapting to the domain

W3: The related works omit some closely related work in the image domain. Benchmarks exist like Roboflow100-VL [1], which has limited training data and covers domains such as medical, aerial, and industrial imagery. For comprehensiveness, the authors should clarify how AnyGroundBench differs from these existing works in the image domain.

Typos in Table 1

  • Caption: “AnyGroundBenchis” > “AnyGroundBench is”
  • Last row: “AnyGroundBench(Ours)” > “AnyGroundBench (Ours)”

[1] Robicheaux et al., Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models, NeurIPS 2025

Questions for authors

Q1: Could the authors confirm the performance gains are not due to additional task-specific data by fine-tuning on HC-STVG, and evaluating on AnyGroundBench?

Q2: Can the authors confirm whether there is leakage in the training/testing splits of the two self-collected benchmarks?

Q3: Can the authors include a brief discussion of the related multi-domain image benchmarks?

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and constructive feedback. We are glad that the reviewers found:

  1. Timely and meaningful benchmark motivation. Evaluating STVG in specialized domains is increasingly necessary (CERu), and providing target-domain training subsets makes the benchmark useful for studying adaptation beyond static zero-shot performance (k6QY, WZQV).
  2. Valuable newly captured, expert-annotated data. The newly collected data is a useful contribution (CERu), and involving medical experts and experienced football players is particularly appropriate (k6QY).
  3. Broad evaluation with diagnostic tasks. The study covers many VLMs and both training-free and fine-tuning-based adaptation (CERu, k6QY), and reporting TVG and SVG alongside STVG reveals differences that an aggregate score would conceal (k6QY).

We summarize the main points addressed in response to the reviews:

  1. SVG/STVG gap (k6QY W3/Q3, WZQV W2/Q2). We add controlled experiments that give SVG the same full-video input as STVG, extract the spatial and temporal components from the same STVG predictions, and compare direct STVG with a sequential TVG→SVG pipeline. The gap persists in all five domains under matched input, and the sequential pipeline outperforms direct STVG in four of five domains.
  2. Splits and model selection (CERu W2/Q2, k6QY W1/Q1–W2/Q2, WZQV W1/Q1). We report the splits of the newly captured data, and a game-disjoint evaluation on American Football yields results close to the original split. No test data are used for any selection.
  3. Domain shift vs. lack of STVG training (CERu W1/Q1). Fine-tuning on general-domain STVG data underperforms fine-tuning on our domain-specific training subsets in all five domains. Together with the sharp drop of models that perform well on general-domain benchmarks, this indicates that the low zero-shot performance stems mainly from domain shift.
  4. Robustness and evaluation details (k6QY W2/Q2, W4/Q4–W6/Q6). We add experiments on training-data budgets, a second backbone (Gemma4-12B), frame sampling rates, fine-tuning seeds, demonstration selection, paired bootstrap CIs, two specialized STVG models, inter-annotator agreement with a human reference, and per-source results. We also specify sparse-to-dense conversion, the treatment of invalid outputs, and the annotation filtering rules. The main findings hold in all cases.

Revisions (all changes are shown in red)

  • Abstract, Section 1, Section 5: updated the number of evaluated VLMs to 25.
  • Section 2: added a discussion of the multi-domain image benchmark Roboflow100-VL [1]. (CERu W3/Q3)
  • Table 1: fixed typos. (CERu)
  • Sections 4.1–4.2, Table 3, Table 5, Table 9, Appendix B, and Appendix B.3: added two specialized STVG models, DeViL [2] and PTD [3]. (k6QY W4/Q4, Q6)
  • Section 3.2, Appendix C (Intended Evaluation Setting), and Appendix I (Generalization Scope): clarified the intended evaluation setting. (k6QY W1/Q1)
  • Appendix C, Appendix H, Tables 16–17: added split details for the newly captured datasets and the game-disjoint evaluation on American Football. (CERu W2/Q2, k6QY W1/Q1, WZQV W1/Q1)
  • Appendix A.4 and Appendix A.6: detailed the GEPA configuration and the adaptation protocol (per-domain/per-task application, fixed configuration, no test-set selection). (k6QY W2/Q2, WZQV W1/Q1)
  • Appendix B (Sparse-to-Dense Conversion and Invalid Predictions): specified tube construction and the scoring of invalid outputs. (k6QY W4/Q4)
  • Appendix C (Sample Filtering and Annotation Rules): stated the criteria for excluding ambiguous samples. (k6QY W5/Q5)
  • Appendix H, Table 15: fine-tuning on HC-STVG-v2 and evaluation on AnyGroundBench. (CERu W1/Q1)
  • Appendix H, Tables 18–20: oracle-controlled experiments and the sequential TVG→SVG baseline. (k6QY W3/Q3, WZQV W2/Q2)
  • Appendix H, Table 21, Figure 12: second backbone (Gemma4-12B) and training-data budgets for SFT LoRA. (k6QY W2/Q2)
  • Appendix H, Table 22: frame sampling rates. (k6QY W4/Q4)
  • Appendix H, Tables 23, 25, 26: fine-tuning seeds, demonstration-selection variability, and paired bootstrap CIs. (k6QY W6/Q6)
  • Appendix H, Tables 27–29: inter-annotator agreement with a human reference, per-source results, and sensitivity with benchmark-wide bins. (k6QY W5/Q5)

We have uploaded the revised paper. If any concern remains, please let us know and we will address it promptly.

[1] Robicheaux et al., Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models, NeurIPS 2025.
[2] Gao et al., Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding, ACM MM 2026.
[3] Rasheed et al., Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding, arXiv 2026.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ends on November 18

Dear Reviewers and Area Chair,

Thank you again for your time and feedback. As the author–reviewer discussion period ends on November 18, we would like to kindly remind you that we have responded to all comments and uploaded a revised paper with all changes shown in red.

We are encouraged that Reviewer CERu noted that "the author response largely addresses" the concerns raised by the other reviewers, and that "AnyGroundBench is a well executed benchmark covering a domain adaptation setting that has not been explored in the past, so it provides real value to the community."

We would greatly appreciate it if you could let us know whether any concerns remain.

AuthorsOpenReview ↗

Summary of Discussion

Dear ACs and Reviewers,

We thank the AC and all reviewers for their insightful comments on our paper. We summarize the discussion with the reviewers.

Reviewer CERu (weak accept). CERu confirmed that their methodological concerns have been addressed, and described AnyGroundBench as “a well executed benchmark covering a domain adaptation setting that has not been explored in the past,” which “provides real value to the community.” The remaining concern raised in the discussion is below.

  • Scope relative to Roboflow100-VL: We clarified that our contribution is to characterize how adaptation behaves on joint spatio-temporal grounding, where TVG/SVG gains transfer poorly to STVG, and we agreed that this gap is a general property of STVG that becomes more severe in specialized domains.

Reviewers k6QY and WZQV. We responded to both reviewers on Nov 13 and followed up twice during the discussion period, but have not yet received responses. Reviewer CERu stated: “I reviewed the concerns raised by other reviewers and believe the author response largely addresses them as well.” We summarize the main concerns of k6QY and WZQV and our responses below:

  • SVG/STVG gap (k6QY W3/Q3, WZQV W2/Q2): With matched full-video input, the gap persists in all five domains, and a sequential TVG→SVG pipeline outperforms direct STVG in four of five (Tables 18–20).

  • Splits and model selection (k6QY W1/Q1–W2/Q2, WZQV W1/Q1): Splits of the newly captured data are now reported, a game-disjoint evaluation on American Football gives results close to the original split, and no test data are used for any selection (Appendix A.4, A.6, C; Tables 16–17).

  • Robustness and evaluation details (k6QY W2/Q2, W4/Q4–W6/Q6): We added a second backbone, training-data budgets, frame sampling rates, fine-tuning seeds, demonstration selection, bootstrap CIs, two specialized STVG models, inter-annotator agreement, and per-source results. The main findings hold in all cases (Tables 3, 5, 9, 21–29). Sparse-to-dense conversion, invalid outputs, and filtering rules are now specified (Appendix B, C).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.