- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Track-MM, an embodied visual tracking framework that explicitly separates appearance and motion information through dual memories. It introduces a Target-Aware Motion Adapter (TMA) to condition pretrained V-JEPA 2.1 features on target representations. A dual-gated memory mechanism separately updates appearance and motion memories using several tracking-related signals. The framework further uses a compact Qwen3-0.6B policy with a conditional flow-matching action head to predict future waypoints. Experiments on EVT-Bench and real-world deployment on a Unitree Go2-W show competitive tracking performance and practical onboard efficiency.
Critical strengths
- Practical Problem. The paper addresses an important problem in embodied visual tracking, particularly under occlusion, target disappearance, and distractors.
- Coherent Architecture. The dual-memory design clearly separates appearance and motion information, while TMA provides target-aware motion features. The overall architecture is coherent.
- Extensive Ablations. The paper provides ablations on motion representations, temporal windows, memory designs, gating strategies, and auxiliary modules. These experiments provide useful evidence for the proposed components.
- Real-World Deployment. The method is deployed on a Unitree Go2-W with a Jetson Orin NX 8GB. The real-world experiments and runtime analysis improve the practical relevance of the work.
Critical weaknesses
-
Limited Methodological Novelty. Most components, including pretrained video representations, target-conditioned adaptation, memory mechanisms, and flow-matching policies, are based on existing techniques. The main novelty appears to lie in their combination rather than a fundamentally new methodology.
-
Limited Baseline Fairness. The main baselines are not retrained under the same training settings. Therefore, the reported improvements may partly result from differences in training data, optimization, or implementation rather than the proposed architecture itself.
-
Reproducibility and Release. The paper does not clearly state whether the training data, data collection/generation code, training code, inference code, evaluation code, and pretrained checkpoints will be fully released. This makes the reproducibility of the reported results difficult to assess.
-
Limited Real-World Evaluation. The real-world evaluation contains 42 trials across four environments. While useful, this scale is still limited for supporting broad claims about robustness across diverse real-world conditions.
-
Incomplete Efficiency Comparison. The paper emphasizes onboard efficiency, but different methods use different hardware and deployment settings. Therefore, the claimed efficiency advantage is difficult to assess under a fully controlled comparison.
Questions for authors
-
What is the key methodological novelty beyond combining pretrained video representations, target-aware adaptation, and memory mechanisms?
-
Can the strongest baselines be retrained using the same training data, evaluation protocol, and computational budget?
-
Will the authors fully release the training data, data collection/generation code, training code, inference code, evaluation code, and pretrained checkpoints? If not, which components will remain unavailable?
-
How are the formulations and thresholds of the gating signals determined? Are they learned or manually designed?
-
How much does each frozen backbone contribute to the final performance? How does the method perform with smaller or simpler alternatives?
4 comments
Q4 - Q5 & Weakness
Q4. How are the gating formulations and thresholds determined?
The gating mechanism combines scores derived from learned representations, manually specified functional forms, and a training-data-calibrated scale; Appendix A.4 summarizes how each component is determined. Here, "learnable" means a score is optimized through its upstream trainable modules, not that its functional form is learned: observability is a trainable projection of the predicted mask; concentration uses a fixed normalized-entropy formula over the response of trainable memory–evidence interactions; continuity uses a fixed exponential-decay form over learned response centroids; and the continuity scale is set to the 95th percentile of target-centroid displacement in the training set. First-step initialization (, clamped to 0.5) is fixed.
Q5. Contribution of each frozen backbone; smaller or simpler alternatives.
We provide complementary evidence. (a) Appearance backbones. We retrained variants without DINOv3 and without SigLIP on DT, with the same data, the same number of epochs, and the same learning rate, adjusting only the input projection for the changed input dimension. Removing DINOv3 reduces SR/TR from 68.9/69.2 to 63.0/60.4 (CR 2.22→3.97); removing SigLIP reduces SR/TR to 64.8/60.0 (CR 2.85); either single-encoder variant remains functional but clearly below the full model under distractors (new table in Appendix A.5). (b) Video backbone. Table 4 compares optical flow and several pretrained video representations across STT/DT/AT, including a simpler optical-flow alternative (SR 66.7 vs. 89.3 on STT); the single-frame configuration drops SR to 77.2, and longer windows do not consistently improve performance. The revised table specifies the backbone variants and parameter counts: VideoMAE2 (ViT-B, ~87M), InternVideo-Next (Large, ~0.3B), V-JEPA 2 (ViT-L, 300M), and V-JEPA 2.1 (ViT-L/16, 300M). (c) Matched backbones. Section 4.5 and Appendix A.9.2 align OmTrackVLA's common pretrained backbones with Track-MM's, reducing backbone-configuration differences in the system-level comparison. (d) Selection rationale and conclusion. The frozen backbones were deliberately chosen at the small end of their respective families (Qwen3-0.6B, DINOv3 ViT-S/16, SigLIP-base, and the ViT-L/16 video encoder), with component sizes listed in Appendix A.4. Taken together, the experiments above suggest that the current combination offers a balanced trade-off between tracking performance and onboard cost.
Update on Weakness 4 (real-world evaluation scale). The revised manuscript expands the real-world evaluation from 42 trials in four environments to 104 trials per method across six environments (shopping mall, urban road, park, office building, parking lot, and outdoor night; Table 2). Track-MM succeeds in 94/104 trials (90.4%) versus 71/104 (68.3%) for OmTrackVLA under the same protocol—identical robot platform, camera, and control stack, with the same designated target and canonical route per trial—yielding a two-sided Fisher exact test of . The largest gains appear in the shopping-mall and outdoor-night scenarios (+30.0 and +28.6 pp). We agree that even this scale remains limited for broad robustness claims; the corresponding limitation, including the need for larger and better-controlled evaluations, is stated in Appendix A.10, and the full protocol (success criteria verified by two independent annotators, failure criteria) together with a failure taxonomy is documented in Appendix A.6.
Update on Weakness 5 (efficiency comparison). The revised manuscript adds matched-hardware measurements at two levels. At the system level, we measured the released ReferTrack checkpoint on the same RTX 3090 with batch size 1 and FP16 inference (137 ms per inference step) versus 55 ms for Track-MM on the same GPU in its edge-cloud mode (Appendix A.9.5), and the base model OmTrackVLA on the same Jetson Orin NX platform (~180 ms; Appendix A.9.2). At the module level, Appendix A.7 reports the latency, parameter-storage, and peak-memory overhead of TMA and the dual-memory module alongside their ablation gains, with the measurement protocol stated explicitly. These additions make the efficiency claims directly checkable under shared hardware, and remaining differences in timing scope and deployment configuration are stated in the corresponding tables.
We thank the reviewer again for the careful assessment and constructive questions, and would be happy to address any further concerns during the discussion period.
Official Comment by Reviewer wgLM
Thank you for the response and the additional experiments. While the revised manuscript adds ablations, baseline comparisons, and a larger real-world evaluation, my main concerns remain unresolved.
First, I am not convinced that the methodological contribution is sufficiently novel. The approach combines pretrained visual and video representations, target-conditioned adaptation, gated memory, and a flow-matching action head. The ablations show that these components are useful, but do not establish a substantial methodological advance beyond their combination.
Second, the baseline comparisons remain insufficiently controlled. Differences in data collection, filtering, auxiliary recognition data, training procedures, and timing scope make it difficult to attribute the reported performance and efficiency gains to the proposed method. The ReferTrack retraining is only an approximate match, and the latency measurements do not use fully comparable timing scopes.
Third, the authors do not clearly state whether the collected training dataset will be released. They plan to release code, checkpoints, and data-generation tools after the review period, but do not specify whether the dataset itself will be available or what materials will remain unavailable. Without access to the data and sufficient provenance and evaluation details, the results cannot be independently verified. This is a serious reproducibility concern in embodied visual tracking, where results depend strongly on the data and evaluation protocol. Given discrepancies between reported metrics and results in some papers in this area, independent verification is particularly important.
The expanded real-world evaluation is useful, but sequential trials and unresolved data availability still limit confidence in the evidence. Overall, my concerns about novelty, baseline fairness, and reproducibility remain. I do not recommend acceptance.
We sincerely thank the reviewer for the continued careful reading and the constructive follow-up. We respect the reviewer's evaluation, but we still want to make the following clarifications.
1. The baseline comparisons
Among the baselines in Table 1, only OmTrackVLA and ReferTrack provide open-source code and released checkpoints. Both Track-MM and ReferTrack derive their tracking data from the EVT-Bench; the differences lie in the post-processing pipelines (expert collection and data filtering). In addition, ReferTrack incorporates QA-based recognition data (stage-1 QA alignment and SYNTH-PEDES refer-QA in stage 2) that our pipeline does not use. We cannot ablate the contribution of another model's auxiliary training data, which is why a fully matched retraining is not feasible; we instead provide the two closest matched comparisons below (in Section 4.5, Appendix A.9.2).
OmTrackVLA (Section 4.5, Appendix A.9.2). We evaluate OmTrackVLA in three configurations — the original checkpoint, the same backbone checkpoints as Track-MM, and a variant additionally retrained on our task-training data — all under the same protocol:
| Method | STT SR / TR / CR | DT SR / TR / CR | AT SR / TR / CR |
|---|---|---|---|
| OmTrackVLA (original) | 81.4 / 82.8 / 5.13 | 41.5 / 58.8 / 11.31 | 60.0 / 73.9 / 7.60 |
| OmTrackVLA (same backbone) | 80.8 / 83.2 / 4.97 | 44.4 / 54.0 / 13.5 | 57.9 / 66.3 / 5.73 |
| OmTrackVLA (same data & backbone) | 82.3 / 80.0 / 3.58 | 38.7 / 38.2 / 20.96 | 61.7 / 55.9 / 5.70 |
| Track-MM (Ours) | 89.3 / 86.7 / 1.19 | 68.9 / 69.2 / 2.22 | 66.8 / 71.4 / 2.64 |
Backbone replacement and matched-data retraining do not consistently improve OmTrackVLA, indicating the gains are attributable to the architecture rather than data or backbone differences.
ReferTrack (Appendix A.9.5). We warm-start from the released stage-1 checkpoint and retrain the stage-2 tracking model on our 12,358-episode tracking data, keeping the official training settings, and evaluate under the same protocol:
| Method | STT SR / TR / CR | DT SR / TR / CR | AT SR / TR / CR |
|---|---|---|---|
| ReferTrack (reported) | 89.4 / 92.5 / 1.6 | 73.3 / 81.8 / 7.6 | 74.1 / 85.7 / 7.7 |
| ReferTrack (retrained on our data) | 88.8 / 87.9 / 1.53 | 72.0 / 76.9 / 5.95 | 70.3 / 74.9 / 8.01 |
| Track-MM (Ours) | 89.3 / 86.7 / 1.19 | 68.9 / 69.2 / 2.22 | 66.8 / 71.4 / 2.64 |
We state the outcome plainly: the retrained ReferTrack remains above Track-MM on DT and AT SR, while Track-MM attains the lowest collision rates in all three settings with 74% fewer parameters (1.2B vs. 4.6B).
2. Release collected training dataset
The training data of the related work in this area, including ours, is all derived from EVT-Bench, and the differences between methods lie only in data processing. The publicly available related work is also the release of data processing tools. We follow the same practice and will release our complete data-processing pipeline, through which the dataset can be obtained.
Thank you again for your careful reading and evaluation. We will continue to improve our paper regardless of the results.
Q1 - Q3
We sincerely thank the reviewer for the careful assessment and the constructive questions. We answer the five questions and close with two updates on the listed weaknesses.
Q1. Key methodological novelty beyond combining existing components.
Our design builds on pretrained video encoders, attention, memory mechanisms, and flow matching; the intended contribution is (a) a way of constructing target-conditioned temporal evidence (TMA), and (b) a way of controlling how evidence sources with asynchronously varying reliability are incorporated into memory. Our design is motivated by the potentially different reliability of appearance and temporal evidence: illumination changes may weaken appearance matching while short-term dynamics remain useful, whereas abrupt target turns or camera motion may reduce the usefulness of continuity cues while identity information remains available. Generic video features also encode distractors, background changes, and camera motion, and therefore benefit from adaptation to the instructed target; TMA provides this via target-importance modulation, bidirectional cross-attention with instruction-conditioned visual features, and auxiliary mask supervision, with the purpose of encouraging target-relevant temporal representations rather than recovering a disentangled physical motion state. On the memory side, appearance and target-conditioned temporal evidence are maintained in two persistent states, and each branch determines its own write strength from memory–evidence response concentration and response-position continuity under a shared observability constraint—providing an explicit way to use different update rates for the two sources within the same observation. We do not claim that a unified memory could not learn similar behavior; our aim is structural support for retaining and adapting the two sources separately. We also acknowledge the connection to prior gated-memory designs such as the TIM of TrackVLA++, which weights updates of selected target features by entropy-based reasoning confidence; our design applies entropy to the spatial memory–evidence correspondence, combined with branch-specific continuity and shared observability.
Q2. Can the strongest baselines be retrained under matched conditions?
Both Track-MM and ReferTrack derive their tracking data from the EVT-Bench train splits, but the two pipelines differ in expert collection and data filtering, and ReferTrack additionally incorporates QA-based recognition stages that our pipeline does not use; a fully matched retraining is therefore not feasible. As the closest matched condition, we retrained its stage-2 model on our 12,358-episode tracking data (warm-started from the released stage-1 checkpoint, official training settings, same evaluation protocol; Appendix A.9.5), obtaining STT/DT/AT SR of 88.8/72.0/70.3, while Track-MM retains the lowest collision rates with 74% fewer parameters. For the base model, Section 4.5 and Appendix A.9.2 evaluate OmTrackVLA under matched backbones and matched training data: Track-MM outperforms all three configurations on SR and CR across STT/DT/AT (e.g., DT SR 68.9 versus at most 44.4). Together with the within-framework ablations (Table 3), these experiments separate the contributions of training data, backbones, and architecture.
Q3. Release of data, code, and checkpoints.
We plan to release the training, inference, and evaluation code, the pretrained checkpoints, and the data-generation and preprocessing tools. Due to institutional approval requirements, these materials cannot be made public during the review period; we aim to release them once the decisions are announced.