acceptodds
Withdrawn from ICLR 2027

CD-RMOT-Bench: Benchmarking Cross-Domain Referring Multi-Object Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Referring multi-object tracking (RMOT) extends tracking from category-driven perception to language-guided understanding by grounding object trajectories in natural-language expressions. Despite recent progress, existing RMOT studies are largely conducted under in-domain settings, leaving the robustness of language-conditioned tracking under inevitable visual domain shifts unexplored. In this paper, we study Cross-Domain Referring Multi-Object Tracking (CD-RMOT), a new and challenging problem that evaluates whether an RMOT model trained on a labeled source domain can reliably follow natural-language expressions in an unlabeled target domain with different visual conditions. To support systematic study, we construct CD-RMOT-Bench, a unified benchmark that combines real clear-domain referring tracking data, aligned digital-twin variants, and real adverse-domain videos. CD-RMOT-Bench enables both controlled weather/viewpoint shift analysis and realistic synthetic-to-real transfer evaluation under a shared RMOT protocol. Further, we provide a Query-Centric Adaptation (QCA) framework for unsupervised domain adaptation (UDA), designed to stabilize the query space that bridges visual trajectories and referring expressions. Extensive experiments reveal that domain shifts severely degrade RMOT performance, affecting both detection and association, with qualitative examples also showing expression-inconsistent target selection. QCA establishes a strong baseline, while CD-RMOT-Bench opens a new direction for robust language-guided tracking across visual domains.

Primary area:
datasets and benchmarks
Keywords:
Referring Multi-Object Tracking, Cross-Domain Tracking, Domain Adaptation, Dataset and Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_1gjZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CD-RMOT-Bench, a benchmark for cross-domain referring multi-object tracking. Its main dataset contribution is Refer-vKITTI, constructed by transferring language–trajectory annotations from five real-world scenes to their synthetic counterparts across ten weather, lighting, and viewpoint conditions. It contains 50 videos, 20,560 frames, and 27,730 expression instances, with 272,043 expression–track pairs retained after filtering. These counts include variants of the same scenes and expressions across domains. Together with the existing Refer-KITTI-V2 and Refer-BDD datasets, the benchmark comprises 121 videos, 38,573 frames, and 42,118 expression instances, supporting controlled domain-shift and synthetic–real transfer evaluation. The paper also proposes QCA, which combines teacher–student consistency, adversarial domain alignment, and language-guided query calibration, and evaluates it across six transfer settings with ablations and multi-seed experiments on selected settings.

Critical strengths

Useful controlled evaluation. Synthetic variants of the same scenes reduce confounding differences in scene content, allowing weather and viewpoint shifts to be studied more directly.Clearly described construction and evaluation protocols. The paper explains identity mapping, visibility filtering, and human auditing, and uses separate scenes for adaptation and evaluation.

Supporting experimental evidence. Experiments cover six transfer settings and include component ablations and three-seed evaluations on two target domains, supporting QCA’s improvements over the source-only baseline.

Critical weaknesses

Insufficient differentiation from recent benchmarks. Table 1 mainly describes the benchmark’s composition rather than establishing its advantages over existing datasets. CRTrack and AerialMind already include cross-domain evaluation, so the additional value of the controlled paired data and unsupervised adaptation protocol should be clarified.Limited independent scene diversity. Refer-vKITTI contains only five underlying scenes, two of which are used for evaluation. Its 50 videos and 27,730 expression instances include repeated variants across domains and therefore do not directly reflect independent visual or linguistic diversity.

Insufficient validation of transferred expressions. Correct identity correspondence does not guarantee that the original descriptions remain valid after viewpoint changes. The audits report annotator agreement but do not quantify semantic correctness across domains and expression types.Confounding factors in comparisons and ablations. The main baselines use source data only, whereas QCA accesses unlabeled target data; the generic adaptation controls also use different training budgets. Additionally, removing CDT-MT stops gradient propagation through ACDA’s target-domain features, making it difficult to isolate individual component contributions.

Questions for authors
  1. Benchmark novelty: How does CD-RMOT-Bench differ from the existing cross-domain evaluation protocols in CRTrack and AerialMind? Could you clarify which additional capabilities arise from the paired scene variants and the target-domain adaptation protocol?
  2. Dataset diversity: How many unique expressions remain after removing repetitions across domain variants? Given that the controlled benchmark uses only five underlying scenes, could you provide per-scene results or an alternative scene-disjoint split to assess sensitivity to the selected training and test scenes?
  3. Annotation validity: How do you verify that transferred expressions, particularly spatial descriptions such as “vehicles on the left,” remain valid after viewpoint changes? Could you report the semantic error rate observed in the 300 audited expression–track instances, together with the correction procedure, rather than annotator agreement alone?
  4. Experimental fairness and component contributions: Could you compare QCA with generic adaptation baselines under the same target-data access and comparable training budgets? Additionally, could you repeat the CDT-MT ablation while preserving ACDA’s target-domain gradient flow, so that the effect of removing CDT-MT can be isolated?
Reviewer_xANjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Cross-Domain Referring Multi-Object Tracking (CD-RMOT), which is a new problem. There are two major contributions: 1. CD-RMOT-Bench, a benchmark that combines real-world referring annotations, aligned synthetic scenes and corrupted videos(weather and viewpoint). 2. Query-Centric Adaptation (QCA) method as a strong baseline of the CD-RMOT problem

Critical strengths
  1. This paper studies a new meaningful problem: Cross-Domain Referring Multi-Object Tracking.
  2. The authors provide detailed annotation-transfer procedures, scene-disjoint evaluation protocols, and annotation quality audits.
Critical weaknesses
  1. The size of this benchmark is relatively small. There are only five aligned scenes. Among these five scenes, three are used as training, only two are used as evaluation. This raises concerns about whether the benchmark can support general conclusions.
  2. A central assumption in the benchmark construction is that referring expressions can be directly transferred across aligned real and synthetic scenes while preserving their original target identities. However, identity correspondence does not necessarily guarantee semantic equivalence across visual domains. First, viewpoint changes of 15° or 30° may alter the referents of spatial expressions such as "cars located on the right,". Second, differences in rendering, illumination, and weather may affect the validity or discriminative power of appearance-based expressions, such as "vehicles in soft tones" (Fig. 4).
Questions for authors

What is the measurable advantage of the proposed aligned digital-twin benchmark over applying standard synthetic weather corruptions to existing RMOT datasets, especially given that the paired domain variants and object annotations are already provided by Virtual KITTI 2?

Reviewer_hye8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-domain referring multi-object tracking, where a model trained with source-domain annotations must track language-specified objects under visual domain shifts. It introduces CD-RMOT-Bench, combining real referring-tracking datasets with a new Refer-vKITTI subset constructed by transferring expressions and trajectories to aligned synthetic weather and viewpoint variants. It also proposes Query-Centric Adaptation (QCA), built on TempRMOT, which combines teacher–student query consistency, curriculum-weighted adversarial domain alignment, and language-guided prototype calibration. Adaptation uses target-domain videos and expressions without tracking annotations. Validation covers six weather, viewpoint, and synthetic–real transfer settings using HOTA, DetA, and AssA, supplemented by component ablations, qualitative examples, and three-seed evaluations on two target domains. The reported results demonstrate substantial degradation under domain shifts and improved tracking performance with QCA relative to source-only baselines and the evaluated adaptation controls.

Critical strengths
  1. The benchmark combines controlled domain shifts with realistic transfer settings. Refer-vKITTI transfers referring annotations to aligned digital-twin variants, enabling weather and viewpoint changes to be studied while limiting changes in scene content. The additional real-domain evaluations complement this controlled setting, and the scene-disjoint evaluation protocol tests generalization beyond adaptation scenes. This design is consequential because it provides a systematic way to investigate cross-domain failures in language-conditioned tracking while retaining a connection to practical deployment conditions.

  2. The proposed adaptation baseline shows consistent benefits across multiple transfer settings. QCA improves HOTA over source-only TempRMOT in all six evaluated transfers. Three-seed experiments on fog and Refer-BDD retain average improvements of approximately 4.8 and 4.7 HOTA points, respectively. These results provide evidence that adaptation benefits extend across both controlled weather shifts and synthetic-to-real transfer, giving the benchmark a useful reference method for subsequent research.

Critical weaknesses
  1. The experiments do not adequately isolate the benefit of the proposed adaptation design. Comparisons against source-only RMOT baselines demonstrate the usefulness of adaptation, but cannot establish the advantage of QCA’s specific mechanisms. The more relevant adaptation controls use separately configured training budgets rather than matched computation, and the CoTTA-style comparator implements only restricted query-state adaptation (Appendix C.2). Moreover, ACDA ablations without CDT-MT detach target features, changing the gradient pathway alongside the module configuration (Appendix B.2). These limitations weaken the evidence that the gains arise from the proposed query-centric mechanisms rather than differences in optimization, training resources, or implementation.

  2. Limited scene diversity and real-world transfer coverage. The controlled subset comprises only five underlying scenes expanded into ten weather/viewpoint variants, with evaluation restricted to two held-out scenes. While this paired construction is useful for controlled analysis, additional variants and transferred expressions do not provide comparable increases in independent scene diversity. The real-domain experiments add useful evidence, but cover synthetic–real transfer without a direct real-to-real adaptation protocol. Consequently, the current evaluation provides limited evidence that the benchmark captures a sufficiently broad range of cross-domain tracking challenges. This is consequential because the benchmark is a principal contribution of the paper.

Questions for authors
  1. Can the authors establish QCA’s advantage through controlled comparisons with matched training budgets and initialization, while keeping target-side gradient flow consistent across component ablations? This would help isolate the contribution of the proposed mechanisms.

  2. Given that the controlled subset contains five underlying scenes and only two evaluation scenes, what evidence shows that the reported gains and method rankings generalize beyond this particular split? Scene-wise results, alternative scene splits, or direct real-to-real transfer results would materially strengthen this assessment.

  3. Can you show that QCA is better at following the text description, beyond improving tracking overall? For example, when the description for the same video changes from “red cars” to “blue cars,” does QCA correctly change which cars it tracks?

Reviewer_Ugh1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CD-RMOT-Bench for cross-domain referring multi-object tracking under a labeled-source, unlabeled-target adaptation protocol. It combines existing real-world resources with Refer-vKITTI, which transfers referring annotations to matched virtual scenes to study weather and viewpoint changes. The paper also proposes QCA on top of TempRMOT, combining teacher-student query consistency, adversarial query alignment, and language-conditioned calibration. Experiments cover controlled domain shifts and transfer to real adverse conditions, with component ablations and three-seed results for two settings. I see the main contribution as the benchmark and evaluation protocol, with QCA providing a useful adaptation baseline.

Critical strengths
  1. The benchmark addresses a practical limitation of evaluating referring tracking only within a single domain. Combining controlled synthetic shifts with a real adverse-domain setting is useful: the former makes particular changes easier to study, while the latter provides a check beyond the simulator. The scene-disjoint evaluation and documented identity-transfer audit are meaningful parts of the contribution.

  2. The experimental study goes beyond comparing QCA only against source-only models. It includes generic adaptation controls, combinations of the proposed components, detection/association breakdowns, and three-seed results on fog and Refer-BDD. These results provide evidence that adaptation helps in the proposed setting and make QCA a useful reference implementation for future work.

Critical weaknesses
  1. The semantic validity of transferred referring expressions needs clearer evidence. Preserving an object's identity across a real scene and its virtual counterpart does not by itself establish that an unchanged expression remains correct after a viewpoint or weather change, particularly for spatial relations, motion descriptions, and appearance attributes. The identity audit and the separate audit of 300 transferred expression-track instances are valuable, but I could not determine the handling rules and validity rates for these potentially affected expression categories under individual shifts. This matters because incorrect or ambiguous expressions could be counted as tracking failures and change the interpretation of the benchmark.

  2. The evidence for the benchmark's breadth and language-specific diagnostic value is still limited. Its controlled synthetic component derives from five base scenes, with two held out for evaluation; the 50 variants should therefore not be interpreted as independent scene diversity. Aggregate HOTA, DetA, and AssA results establish overall robustness differences, but do not separate failures in language-based target selection from ordinary detection or association degradation. The qualitative examples are useful, yet leave uncertainty about how widely the observed language-related effects occur. This limits the strength of the conclusions that can currently be drawn from the benchmark, rather than negating the value of the resource.

Questions for authors
  1. How are expressions whose truth may change with viewpoint or weather handled during transfer? For the 300-instance audit, could you clarify the semantic acceptance criteria, the number of invalid or ambiguous instances, and the breakdown by expression type and domain shift? In particular, were spatial relations and appearance-dependent expressions checked separately from identity correspondence?

  2. Do the existing results allow a per-scene or expression-type breakdown that distinguishes language-based target-selection failures from general visual tracking degradation? This would help assess whether the benchmark supports language-specific conclusions and how sensitive the reported gains are to the two held-out synthetic scenes.

  3. Could you sharpen the distinction from prior cross-view RMOT work such as CRTrack, which already considers cross-domain evaluation? My understanding is that the distinctive contribution here is controlled weather/viewpoint shifts and the labeled-source, unlabeled-target adaptation protocol, rather than cross-domain RMOT in general. Please clarify this scope.

  4. Table 3 provides useful generic adaptation controls, but uses single runs and different training budgets. Are comparable-budget or repeated-run results available for the strongest control? If not, please clarify how much of QCA's advantage can be attributed to its specific components beyond the overall adaptation and optimization budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.