acceptodds
Under review as a conference paper at ICLR 2027

Track-MM: Dual-Memory Embodied Visual Tracking with Latent Motion Priors

Abstract

Embodied visual tracking (EVT) is an essential capability for embodied agents, which needs persistent target tracking in dynamic, partially observable environments. The core challenge lies in maintaining target identity under distraction with limited computing and memory resources. To address the challenges, we propose Track-MM, a lightweight dual-memory framework for robust embodied visual tracking. Track-MM builds target-aware latent motion priors by leveraging pre-trained video foundation model representations via a lightweight target-aware motion adapter. The framework separately maintains appearance and motion memories with independent update gates to adapt to dynamic observations while retaining target identity. Benefiting from the lightweight design, Track-MM runs on resource-limited platforms with reduced parameters than mainstream methods. Evaluations on EVT-Bench demonstrate that our method strikes a favorable balance between tracking performance and computational efficiency. Real-world experiments further demonstrate promising zero-shot sim-toreal transfer, where the method maintains reliable tracking under varied lighting conditions on NVIDIA Orin platforms with practical deployment efficiency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Visual Tracking, Vision-Language-Action Models, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wgLMOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Track-MM, an embodied visual tracking framework that explicitly separates appearance and motion information through dual memories. It introduces a Target-Aware Motion Adapter (TMA) to condition pretrained V-JEPA 2.1 features on target representations. A dual-gated memory mechanism separately updates appearance and motion memories using several tracking-related signals. The framework further uses a compact Qwen3-0.6B policy with a conditional flow-matching action head to predict future waypoints. Experiments on EVT-Bench and real-world deployment on a Unitree Go2-W show competitive tracking performance and practical onboard efficiency.

Critical strengths
  1. Practical Problem. The paper addresses an important problem in embodied visual tracking, particularly under occlusion, target disappearance, and distractors.
  2. Coherent Architecture. The dual-memory design clearly separates appearance and motion information, while TMA provides target-aware motion features. The overall architecture is coherent.
  3. Extensive Ablations. The paper provides ablations on motion representations, temporal windows, memory designs, gating strategies, and auxiliary modules. These experiments provide useful evidence for the proposed components.
  4. Real-World Deployment. The method is deployed on a Unitree Go2-W with a Jetson Orin NX 8GB. The real-world experiments and runtime analysis improve the practical relevance of the work.
Critical weaknesses
  1. Limited Methodological Novelty. Most components, including pretrained video representations, target-conditioned adaptation, memory mechanisms, and flow-matching policies, are based on existing techniques. The main novelty appears to lie in their combination rather than a fundamentally new methodology.

  2. Limited Baseline Fairness. The main baselines are not retrained under the same training settings. Therefore, the reported improvements may partly result from differences in training data, optimization, or implementation rather than the proposed architecture itself.

  3. Reproducibility and Release. The paper does not clearly state whether the training data, data collection/generation code, training code, inference code, evaluation code, and pretrained checkpoints will be fully released. This makes the reproducibility of the reported results difficult to assess.

  4. Limited Real-World Evaluation. The real-world evaluation contains 42 trials across four environments. While useful, this scale is still limited for supporting broad claims about robustness across diverse real-world conditions.

  5. Incomplete Efficiency Comparison. The paper emphasizes onboard efficiency, but different methods use different hardware and deployment settings. Therefore, the claimed efficiency advantage is difficult to assess under a fully controlled comparison.

Questions for authors
  1. What is the key methodological novelty beyond combining pretrained video representations, target-aware adaptation, and memory mechanisms?

  2. Can the strongest baselines be retrained using the same training data, evaluation protocol, and computational budget?

  3. Will the authors fully release the training data, data collection/generation code, training code, inference code, evaluation code, and pretrained checkpoints? If not, which components will remain unavailable?

  4. How are the formulations and thresholds of the gating signals determined? Are they learned or manually designed?

  5. How much does each frozen backbone contribute to the final performance? How does the method perform with smaller or simpler alternatives?

4 comments

We sincerely thank the reviewer for the careful assessment and the constructive questions. We answer the five questions and close with two updates on the listed weaknesses.

Q1. Key methodological novelty beyond combining existing components.

Our design builds on pretrained video encoders, attention, memory mechanisms, and flow matching; the intended contribution is (a) a way of constructing target-conditioned temporal evidence (TMA), and (b) a way of controlling how evidence sources with asynchronously varying reliability are incorporated into memory. Our design is motivated by the potentially different reliability of appearance and temporal evidence: illumination changes may weaken appearance matching while short-term dynamics remain useful, whereas abrupt target turns or camera motion may reduce the usefulness of continuity cues while identity information remains available. Generic video features also encode distractors, background changes, and camera motion, and therefore benefit from adaptation to the instructed target; TMA provides this via target-importance modulation, bidirectional cross-attention with instruction-conditioned visual features, and auxiliary mask supervision, with the purpose of encouraging target-relevant temporal representations rather than recovering a disentangled physical motion state. On the memory side, appearance and target-conditioned temporal evidence are maintained in two persistent states, and each branch determines its own write strength from memory–evidence response concentration and response-position continuity under a shared observability constraint—providing an explicit way to use different update rates for the two sources within the same observation. We do not claim that a unified memory could not learn similar behavior; our aim is structural support for retaining and adapting the two sources separately. We also acknowledge the connection to prior gated-memory designs such as the TIM of TrackVLA++, which weights updates of selected target features by entropy-based reasoning confidence; our design applies entropy to the spatial memory–evidence correspondence, combined with branch-specific continuity and shared observability.

Q2. Can the strongest baselines be retrained under matched conditions?

Both Track-MM and ReferTrack derive their tracking data from the EVT-Bench train splits, but the two pipelines differ in expert collection and data filtering, and ReferTrack additionally incorporates QA-based recognition stages that our pipeline does not use; a fully matched retraining is therefore not feasible. As the closest matched condition, we retrained its stage-2 model on our 12,358-episode tracking data (warm-started from the released stage-1 checkpoint, official training settings, same evaluation protocol; Appendix A.9.5), obtaining STT/DT/AT SR of 88.8/72.0/70.3, while Track-MM retains the lowest collision rates with 74% fewer parameters. For the base model, Section 4.5 and Appendix A.9.2 evaluate OmTrackVLA under matched backbones and matched training data: Track-MM outperforms all three configurations on SR and CR across STT/DT/AT (e.g., DT SR 68.9 versus at most 44.4). Together with the within-framework ablations (Table 3), these experiments separate the contributions of training data, backbones, and architecture.

Q3. Release of data, code, and checkpoints.

We plan to release the training, inference, and evaluation code, the pretrained checkpoints, and the data-generation and preprocessing tools. Due to institutional approval requirements, these materials cannot be made public during the review period; we aim to release them once the decisions are announced.

AuthorsOpenReview ↗

Q4 - Q5 & Weakness

Q4. How are the gating formulations and thresholds determined?

The gating mechanism combines scores derived from learned representations, manually specified functional forms, and a training-data-calibrated scale; Appendix A.4 summarizes how each component is determined. Here, "learnable" means a score is optimized through its upstream trainable modules, not that its functional form is learned: observability is a trainable projection of the predicted mask; concentration uses a fixed normalized-entropy formula over the response of trainable memory–evidence interactions; continuity uses a fixed exponential-decay form over learned response centroids; and the continuity scale dmax⁡d_{\max} is set to the 95th percentile of target-centroid displacement in the training set. First-step initialization (qb1=1q_b^1{=}1, sb1s_b^1 clamped to 0.5) is fixed.

Q5. Contribution of each frozen backbone; smaller or simpler alternatives.

We provide complementary evidence. (a) Appearance backbones. We retrained variants without DINOv3 and without SigLIP on DT, with the same data, the same number of epochs, and the same learning rate, adjusting only the input projection for the changed input dimension. Removing DINOv3 reduces SR/TR from 68.9/69.2 to 63.0/60.4 (CR 2.22→3.97); removing SigLIP reduces SR/TR to 64.8/60.0 (CR 2.85); either single-encoder variant remains functional but clearly below the full model under distractors (new table in Appendix A.5). (b) Video backbone. Table 4 compares optical flow and several pretrained video representations across STT/DT/AT, including a simpler optical-flow alternative (SR 66.7 vs. 89.3 on STT); the single-frame configuration drops SR to 77.2, and longer windows do not consistently improve performance. The revised table specifies the backbone variants and parameter counts: VideoMAE2 (ViT-B, ~87M), InternVideo-Next (Large, ~0.3B), V-JEPA 2 (ViT-L, 300M), and V-JEPA 2.1 (ViT-L/16, 300M). (c) Matched backbones. Section 4.5 and Appendix A.9.2 align OmTrackVLA's common pretrained backbones with Track-MM's, reducing backbone-configuration differences in the system-level comparison. (d) Selection rationale and conclusion. The frozen backbones were deliberately chosen at the small end of their respective families (Qwen3-0.6B, DINOv3 ViT-S/16, SigLIP-base, and the ViT-L/16 video encoder), with component sizes listed in Appendix A.4. Taken together, the experiments above suggest that the current combination offers a balanced trade-off between tracking performance and onboard cost.

Update on Weakness 4 (real-world evaluation scale). The revised manuscript expands the real-world evaluation from 42 trials in four environments to 104 trials per method across six environments (shopping mall, urban road, park, office building, parking lot, and outdoor night; Table 2). Track-MM succeeds in 94/104 trials (90.4%) versus 71/104 (68.3%) for OmTrackVLA under the same protocol—identical robot platform, camera, and control stack, with the same designated target and canonical route per trial—yielding a two-sided Fisher exact test of p≈1.3×10−4p\approx1.3\times10^{-4}. The largest gains appear in the shopping-mall and outdoor-night scenarios (+30.0 and +28.6 pp). We agree that even this scale remains limited for broad robustness claims; the corresponding limitation, including the need for larger and better-controlled evaluations, is stated in Appendix A.10, and the full protocol (success criteria verified by two independent annotators, failure criteria) together with a failure taxonomy is documented in Appendix A.6.

Update on Weakness 5 (efficiency comparison). The revised manuscript adds matched-hardware measurements at two levels. At the system level, we measured the released ReferTrack checkpoint on the same RTX 3090 with batch size 1 and FP16 inference (137 ms per inference step) versus 55 ms for Track-MM on the same GPU in its edge-cloud mode (Appendix A.9.5), and the base model OmTrackVLA on the same Jetson Orin NX platform (~180 ms; Appendix A.9.2). At the module level, Appendix A.7 reports the latency, parameter-storage, and peak-memory overhead of TMA and the dual-memory module alongside their ablation gains, with the measurement protocol stated explicitly. These additions make the efficiency claims directly checkable under shared hardware, and remaining differences in timing scope and deployment configuration are stated in the corresponding tables.

We thank the reviewer again for the careful assessment and constructive questions, and would be happy to address any further concerns during the discussion period.

Reviewer_wgLMOpenReview ↗

Official Comment by Reviewer wgLM

Thank you for the response and the additional experiments. While the revised manuscript adds ablations, baseline comparisons, and a larger real-world evaluation, my main concerns remain unresolved.

First, I am not convinced that the methodological contribution is sufficiently novel. The approach combines pretrained visual and video representations, target-conditioned adaptation, gated memory, and a flow-matching action head. The ablations show that these components are useful, but do not establish a substantial methodological advance beyond their combination.

Second, the baseline comparisons remain insufficiently controlled. Differences in data collection, filtering, auxiliary recognition data, training procedures, and timing scope make it difficult to attribute the reported performance and efficiency gains to the proposed method. The ReferTrack retraining is only an approximate match, and the latency measurements do not use fully comparable timing scopes.

Third, the authors do not clearly state whether the collected training dataset will be released. They plan to release code, checkpoints, and data-generation tools after the review period, but do not specify whether the dataset itself will be available or what materials will remain unavailable. Without access to the data and sufficient provenance and evaluation details, the results cannot be independently verified. This is a serious reproducibility concern in embodied visual tracking, where results depend strongly on the data and evaluation protocol. Given discrepancies between reported metrics and results in some papers in this area, independent verification is particularly important.

The expanded real-world evaluation is useful, but sequential trials and unresolved data availability still limit confidence in the evidence. Overall, my concerns about novelty, baseline fairness, and reproducibility remain. I do not recommend acceptance.

We sincerely thank the reviewer for the continued careful reading and the constructive follow-up. We respect the reviewer's evaluation, but we still want to make the following clarifications.

1. The baseline comparisons

Among the baselines in Table 1, only OmTrackVLA and ReferTrack provide open-source code and released checkpoints. Both Track-MM and ReferTrack derive their tracking data from the EVT-Bench; the differences lie in the post-processing pipelines (expert collection and data filtering). In addition, ReferTrack incorporates QA-based recognition data (stage-1 QA alignment and SYNTH-PEDES refer-QA in stage 2) that our pipeline does not use. We cannot ablate the contribution of another model's auxiliary training data, which is why a fully matched retraining is not feasible; we instead provide the two closest matched comparisons below (in Section 4.5, Appendix A.9.2).

OmTrackVLA (Section 4.5, Appendix A.9.2). We evaluate OmTrackVLA in three configurations — the original checkpoint, the same backbone checkpoints as Track-MM, and a variant additionally retrained on our task-training data — all under the same protocol:

MethodSTT SR / TR / CRDT SR / TR / CRAT SR / TR / CR
OmTrackVLA (original)81.4 / 82.8 / 5.1341.5 / 58.8 / 11.3160.0 / 73.9 / 7.60
OmTrackVLA (same backbone)80.8 / 83.2 / 4.9744.4 / 54.0 / 13.557.9 / 66.3 / 5.73
OmTrackVLA (same data & backbone)82.3 / 80.0 / 3.5838.7 / 38.2 / 20.9661.7 / 55.9 / 5.70
Track-MM (Ours)89.3 / 86.7 / 1.1968.9 / 69.2 / 2.2266.8 / 71.4 / 2.64

Backbone replacement and matched-data retraining do not consistently improve OmTrackVLA, indicating the gains are attributable to the architecture rather than data or backbone differences.

ReferTrack (Appendix A.9.5). We warm-start from the released stage-1 checkpoint and retrain the stage-2 tracking model on our 12,358-episode tracking data, keeping the official training settings, and evaluate under the same protocol:

MethodSTT SR / TR / CRDT SR / TR / CRAT SR / TR / CR
ReferTrack (reported)89.4 / 92.5 / 1.673.3 / 81.8 / 7.674.1 / 85.7 / 7.7
ReferTrack (retrained on our data)88.8 / 87.9 / 1.5372.0 / 76.9 / 5.9570.3 / 74.9 / 8.01
Track-MM (Ours)89.3 / 86.7 / 1.1968.9 / 69.2 / 2.2266.8 / 71.4 / 2.64

We state the outcome plainly: the retrained ReferTrack remains above Track-MM on DT and AT SR, while Track-MM attains the lowest collision rates in all three settings with 74% fewer parameters (1.2B vs. 4.6B).

2. Release collected training dataset

The training data of the related work in this area, including ours, is all derived from EVT-Bench, and the differences between methods lie only in data processing. The publicly available related work is also the release of data processing tools. We follow the same practice and will release our complete data-processing pipeline, through which the dataset can be obtained.

Thank you again for your careful reading and evaluation. We will continue to improve our paper regardless of the results.

Reviewer_jz7mOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Track-MM focuses on the identity confusion caused by occlusion, similar targets, and illumination changes in embodied visual tracking, as well as the problem that large models are difficult to be deployed on the robot side. The authors propose a target-aware motion adapter, which separately maintains appearance and motion memories. Through independent gating control for updates, and combined with language instructions to predict the tracking route. Experiments show that this method achieves competitive performance on EVT-Bench and can also run on edge devices; ablation experiments support the role of the main modules. However, due to the inconsistent training data and base of the comparison method, as well as the limited scale of real robot tests, its relative advantages and generalization ability still need more rigorous verification.

Critical strengths
  1. Separating the target appearance and motion cues helps maintain the target identity in the presence of similar pedestrians.
  2. The appearance and motion memories have their own gating mechanisms, which can reduce errors caused by occlusions or unreliable images.
  3. The system with approximately 1.2 billion parameters can run completely natively on the Jetson Orin NX. The paper also provides real robot test results.
Critical weaknesses
  1. Table 1 employs a unified testing protocol, but the training data, pre-trained models, and base sizes of each method are different, making it impossible to directly attribute the performance differences to the dual-memory design.
  2. The paper reported the running speed of the machine and the module latency, but did not measure the main competing methods under the same hardware and inference settings, nor did it present the performance gains and latency, as well as the memory overhead, of the newly added modules.
Questions for authors
  1. For the main benchmark models in Table 1, could you please explain their training data, pre-trained backbone networks, and the number of parameters? Is there a comparison using the same training data and backbone networks to help isolate the contribution of the dual-memory design?
  2. Does the strongest benchmark model provide matching hardware configuration and measurement results under the inference environment? Could you also report separately the increase in inference latency and peak memory usage for TMA and the dual-memory module, along with the performance improvements brought about by them?
7 comments

We sincerely thank the reviewer for the positive assessment and for the constructive questions, both of which prompted substantial additions to the revised appendix. We summarize these below.

Q1 — Training data, pretrained backbones, and parameter counts of the Table 1 baselines.

We have added a complete accounting in Appendix A.9.1, in two tables. The first lists the task-training data of each method (foundation-model pretraining corpora excluded): for example, TrackVLA++ uses 1M Polar-CoT tracking and 1M QA samples, and ReferTrack is trained in two stages (multimodal QA alignment, followed by 1.3M tracking and 1.3M Refer-QA samples). The second lists the pretrained components and complete-system parameter counts: for example, ReferTrack (Qwen3-4B, DINOv3 ViT-S/16, SigLIP SO400M, YOLO11x; ≈4.6B) and Track-MM (Qwen3-0.6B, DINOv3, SigLIP-base, V-JEPA 2.1, Grounding-DINO attention; ≈1.2B, of which ~63M trainable). We also document the provenance of every Table 1 number—quoted from original publications or official releases versus evaluated by us from released checkpoints—in Appendix A.9.7, since comparability depends on this information as well.

Q2 — Matched training data and backbone comparison isolating the dual-memory design.

We provide two complementary matched experiments. (a) Matched base model (Section 4.5 and Appendix A.9.2). Track-MM is built on OmTrackVLA, so we evaluate OmTrackVLA under three configurations: its original checkpoint; the same backbone checkpoints as Track-MM ("same backbone"); and additionally retrained on the same task-training data ("same data and backbone"). Track-MM outperforms all three on SR and CR across STT, DT, and AT (e.g., DT SR 68.9 versus at most 44.4). (b) Strongest baseline under our data (Appendix A.9.5). We retrain ReferTrack on our 12,358-episode tracking data, warm-started from its released stage-1 checkpoint with the official training settings; it reaches STT/DT/AT SR of 88.8/72.0/70.3, while Track-MM retains the lowest collision rates in all settings with 74% fewer parameters.

Together, these matched comparisons remove the training-data and backbone confounds, and the within-framework ablation (Table 3) isolates the dual-memory design specifically: replacing the separate memories with a unified memory, or the branch-specific gates with a shared gate, consistently degrades performance—the best unified-memory variant reaches DT/AT SR of 65.6/60.1 versus 68.9/66.8 for the full model, and removing memory entirely drops AT SR to 50.7.

Q3 — Matched-hardware latency for the strongest baseline.

We measured the released ReferTrack checkpoint on the same RTX 3090 workstation with batch size 1 and FP16 inference, obtaining 137 ms per inference step, versus 55 ms for Track-MM on the same GPU in its edge-cloud mode (including communication overhead). We also measured the base model OmTrackVLA on the same Jetson platform, so the edge-side comparison with the base model uses matched hardware:

MethodHardwareLatency (ms)
ReferTrack (released, FP16, batch 1)RTX 3090137
Track-MM (edge-cloud mode, FP16, incl. comm.)RTX 309055
OmTrackVLA (our deployment config.)Jetson Orin NX180
Track-MM (fully onboard)Jetson Orin NX161

A fuller account is provided in the appendix: Appendix A.9.5 reports the measurement setup and the caveat that the 137 ms vs. 55 ms comparison is a same-GPU reference with different timing scopes (model latency only vs. edge-cloud incl. communication), and the onboard figures are same-scope end-to-end; Appendix A.9.2 analyzes the source of Track-MM's additional computation relative to the shared Qwen3-0.6B backbone; and Appendix A.7 gives the module-level latency and memory breakdown of the two added modules.

Q4 — Latency and memory overhead of TMA and the dual-memory module, together with their gains.

Appendix A.7 reports the performance gains and measured deployment overhead of both modules on the Jetson Orin NX (20 W mode, batch size 1, FP16 inference), summarized below:

ModuleDT SR gain (pp)AT SR gain (pp)Latency increase (ms/step)Peak-memory increase (MiB)
TMA+12.5+4.52.1–6.542–48
Dual memory+7.1+16.11.0–3.39–12

The latency overhead of each module is the per-step difference between the full model and the corresponding ablated variant, reported as the minimum–maximum over consecutive inference steps after 50 warm-up steps. Together, the two modules add at most ~10 ms to the 161 ms end-to-end latency (<7% overhead).

We thank the reviewer again for the positive assessment and constructive questions, and would be happy to discuss any further points during the discussion period.

First, the authors’ response does not clearly map to my questions, which makes it difficult to follow. In addition, the 1.2B model does not achieve consistently leading performance. Why did the authors not also evaluate a larger model to enable a fairer comparison with the baselines?

We apologize for the unclear structure of our previous reply. The reviewer's two questions each contained two sub-questions, which we answered separately as four points. The correspondence is:

  • Your Question 1 (training data, backbones, and parameter counts; the matched-data/backbone comparison) → our reply's Q1 and Q2.
  • Your Question 2 (matched-hardware measurements; module-level overhead with gains) → our reply's Q3 and Q4.

To avoid further confusion, the answer below follows the order of your original review. For convenience, we also reproduce the key tables from the appendix directly below, so the answer can be read without consulting the revised PDF.

Why we did not evaluate a larger model.

  1. Running under tight resource budgets is a design goal. This is a practical consideration rather than a preference for small models. Based on our investigation and on observing current industry practice, deploying large models in embodied tracking involves persistent challenges: (i) the network link becomes a single point of failure for outdoor or indoor scenes with unstable connectivity; (ii) end-to-end latency depends on bandwidth conditions rather than the robot itself; and (iii) continuously streaming videos of pedestrians raises privacy concerns in public environments, where robots may not retain coverage of high-bandwidth connectivity in the first place.

  2. Fixing the scale is what isolates the design. The reviewer's first weakness notes that differences in training data, backbones, and sizes prevent attributing gains to the dual-memory design. That is precisely why we kept the scale fixed and showed the gain survives matched data and matched backbones (DT SR 68.9 versus at most 44.4, Section 4.5): with the scale held constant, the remaining gap is attributable to the added modules---namely the video branch, TMA, and the dual memory. Scaling the backbone simultaneously would reintroduce the confound the reviewer points to.

  3. We will do our best to conduct an experiment to replace the larger base model before the discussion ends.

Reviewer_rKVyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Track-MM extends OmTrackVLA for embodied visual tracking. A frozen V-JEPA 2.1 encodes the last four frames. A target-aware motion adapter fuses these tokens with Grounding DINO text-image attention, and a mask head supervises the result. Appearance and motion features are stored in separate memories. Each memory has its own update gate, computed as the product of visibility, response concentration and position continuity. A flow-matching head predicts the waypoints. With 63M trainable parameters, Track-MM has the lowest collision rate on EVT-Bench. It is also deployed zero-shot on a quadruped robot with a Jetson Orin NX.

Critical strengths
  1. The policy runs fully onboard on an edge device and is compared with its base model on the same robot. It also has the lowest collision rate in all simulated settings.

  2. The ablations cover the memory structure, each gate factor and the choice of video encoder.

Critical weaknesses
  1. The motivation and novelty of the dual memory are not established. TrackVLA++, which the paper cites, already uses a confidence-gated target memory to prevent corruption from distractors or target absence, but the paper does not discuss the difference. It is also not shown that the motion branch encodes motion. Fmot is trained to predict the target mask, and in Figures 3 and 6 it looks like a target localization map. It may therefore act as a second localization branch rather than a motion prior. Moreover, both gates share the visibility term, so both close when the target is not visible. The motion memory therefore cannot provide continuity during occlusion, which is the role the introduction assigns to it.

  2. The main comparison leaves out the base model and is not consistent. OmTrackVLA, on which Track-MM is built, is not in Table 1. ReferTrack is higher in SR and TR in all three settings, yet Figure 8 labels Track-MM as having the highest DT success rate. The appendix also states that all baselines were re-evaluated under the same protocol. However, the Table 1 numbers appear to match the original papers, and VLingNav has no AT result. The evaluation covers only EVT-Bench, whose scenes come from the same simulator as the training data. TrackVLA and TrackVLA++ additionally report zero-shot results on Gym-UnrealCV, which would better support the generalization claim.

Questions for authors
  1. How does the proposed memory differ from the gated Target Identification Memory of TrackVLA++?

  2. Is there evidence that Fmot encodes target motion rather than target location? How does the motion memory help when the target is occluded and both gates are closed?

  3. What are the full results of OmTrackVLA in Table 1? Were the baseline numbers re-evaluated or taken from the original papers?

  4. Why are the component ablations reported only on DT and AT, while the video encoder and window length are chosen only on STT?

  5. How does Track-MM perform zero-shot on Gym-UnrealCV?

5 comments

We sincerely thank the reviewer for the careful reading and constructive feedback. We address each concern below and all referenced changes are included in the revised manuscript.

Q1. Difference from TrackVLA++'s gated Target Identification Memory (TIM)

We agree that confidence-gated memory updates are not first introduced by our work—TIM in TrackVLA++ is a prominent example—and that the submitted manuscript should have discussed this connection explicitly. We clarify the differences below.

The shared principle is that both methods suppress memory writes when the evidence for the target is unreliable. Beyond this, the designs differ in the state maintained and in how the gate is computed:

(i) TIM maintains a single long-term identity memory. It stores visual features selected by a Polar-CoT reasoning token, and weights each update by that token's confidence—computed as the normalized entropy of its logits over the reasoning vocabulary, normalized against its historical average—with an <invalid> token forcing the memory to freeze. TrackVLA++ pairs TIM with TrackVLA's sliding visual window as its "dual-memory" design.

(ii) Track-MM maintains two recurrent states holding distinct evidence types (appearance and target-conditioned video-temporal features), each with its own write control, because the reliability of the two sources need not change together: for example, low illumination can degrade appearance matching while short-term dynamics remain informative; abrupt target turns, camera shake, or ego-motion can weaken continuity cues while appearance identity remains reliable; and with visually similar distractors, appearance matching is ambiguous while distinct motion patterns remain discriminative.

(iii) Our gate signals are computed from memory–evidence interaction statistics—spatial response concentration and response-centroid continuity—under a shared observability prior, rather than from the confidence of a language-model reasoning token.

We acknowledge that the normalized-entropy form of our concentration term is inspired by TIM's confidence computation; it operates on a different object (the spatial memory–evidence response distribution rather than reasoning-token logits) and serves as one of three branch-specific factors rather than the sole update weight. The unified/separate-state and shared/branch-specific-gate ablations (Table 3) support the value of these choices within our framework; they are not a controlled head-to-head comparison with TIM.

Q2. Does FmotF_\mathrm{mot} encode motion rather than location?

We have added a controlled input comparison in Appendix A.2 (the paragraph "Motion-related information in FmotF_\mathrm{mot}", Figure 7). Using the same model, we compare a temporally ordered video with a sequence built by repeating its final frame; both inputs share the same length and final observation, with all other settings fixed. In the first example, the target becomes almost entirely invisible in the final frame: sequential input retains a response near where the target leaves view, whereas repeated-frame input produces a prominent response on unrelated background. In the second, two visually identical people are present, one moving and one stationary: sequential input responds selectively to the moving target, whereas repeated-frame input responds to both. Since the final observation is identical in each pair, these differences support the presence of motion-related information in FmotF_\mathrm{mot} beyond current-frame localization. We regard this as qualitative supporting evidence—not proof of a disentangled physical motion state—and state this scope explicitly in the appendix.

Q3. Motion memory under occlusion, when both gates close

Suppressing memory writes does not disable memory reads: when the shared observability score is low, both branches stop incorporating unreliable current observations, while their accumulated states remain available to the policy for action prediction and target re-association upon reappearance. The accumulated temporal context thus supports recovery, in the sense of retaining useful history rather than extrapolating an invisible target. We have refined the wording in Section 1 accordingly.

Q4. OmTrackVLA, the Pareto figure, and evaluation provenance

First, we have added OmTrackVLA to Table 1 with complete STT/DT/AT results, and documented its configuration, checkpoint, and evaluation settings in Appendix A.9.2. In the submission we quoted published baselines and did not list it separately, since its results are reported in an official repository rather than a paper; we agree that, as the implementation Track-MM is built on, it is an essential reference. Section 4.5 and Appendix A.9.2 further provide matched-data and matched-backbone comparisons showing that Track-MM's advantage is not explained by training data or backbone differences alone.

Second, we have corrected the Pareto figure (Fig. 8 in the submission) and its discussion. The revised figure includes ReferTrack (4.6B, DT SR 73.3%), which is higher than Track-MM (1.2B, 68.9%); we now describe our results as competitive tracking performance at a smaller scale with the lowest collision rates, rather than the highest success rate.

Third, we have audited the provenance of all baseline numbers. All results in Table 1 are quoted from the original publications or official releases; our own evaluations of released checkpoints (OmTrackVLA, and a ReferTrack retraining experiment) are reported separately in the appendix. The earlier statement that all baselines were re-evaluated under identical settings was incorrect and has been removed (Appendix A.9.7); for quoted results we no longer imply matching evaluation seeds or implementation details. VLingNav's code and checkpoints were not publicly available at the time of our evaluation, so we quote its published numbers; its AT entries are left unreported because the original paper does not report them.

Q5. Ablation coverage

We initially prioritized DT/AT for component ablations because distractors and appearance ambiguity most directly challenge identity preservation, and STT for encoder/window selection. We agree this asymmetry does not establish that the choices transfer across settings. In the revision, Table 3 reports component ablations on STT/DT/AT, and Table 4 reports video-encoder and window-length experiments on all three settings. The selected configuration (V-JEPA 2.1, K=4) achieves the best SR on STT and DT and the best CR on all three settings; longer windows improve some metrics (e.g., STT/AT TR at K=16) without consistent gains.

Q6. Zero-shot evaluation on Gym-UnrealCV

Following the reviewer's suggestion, we have added zero-shot evaluation on Gym-UnrealCV in Appendix A.3 (Table 6, with qualitative results), following the TrackVLA++ evaluation settings for Single Target, Distractor, and Unseen Objects, without any training or fine-tuning on Gym-UnrealCV data. We report episode length (EL, up to 500 steps) and success rate (SR; an episode succeeds if it completes the full 500 steps); baseline numbers are quoted from TrackVLA++, as indicated in the caption. Track-MM reaches 500/1.00 EL/SR on Single Target and Unseen Objects, and 496/0.99 on Distractor, improving over the reported TrackVLA++ results (484/0.92) under distractor interference. Together with the zero-shot real-world deployment (Table 2), this provides cross-simulator and sim-to-real evidence beyond EVT-Bench.

We thank the reviewer again for the careful reading and constructive feedback, and would be happy to address any further concerns during the discussion period.

Thank you for the detailed response and the additional experiments. I have read the rebuttal and the revised manuscript, although the changes are not marked, which makes them hard to locate and verify. Two concerns remain.

  1. Motion information in FmotF_{mot}. The evidence that FmotF_{mot} encodes motion is still limited to two qualitative examples (Appendix A.2, Figure 7). A quantitative evaluation is needed, for example the benchmark results when FmotF_{mot} receives the repeated-frame input.

  2. Advantage over OmTrackVLA. Compared with OmTrackVLA, Track-MM keeps DINOv3 and Qwen3-0.6B, replaces SigLIP SO400M with SigLIP-base, and adds V-JEPA 2.1 and the Grounding DINO text–image attention (Table 20). The backbone replacement alone does not consistently improve OmTrackVLA (Table 5), and the paper does not show why the added modules are preferable to a simpler design. The paper emphasizes computational efficiency, but the method adds modules to OmTrackVLA rather than simplifying it, and its size (about 1.2B) is larger than that of OmTrackVLA (about 1.1B). Table 15 does not compare the onboard FPS with OmTrackVLA, the base model of similar size, so the efficiency advantage of the proposed method is not demonstrated.

I will therefore keep my rating.

Comments on the paper

2 comments
Area_Chair_M39BOpenReview ↗

AC Reminder: Please Confirm Whether Your Concerns Are Addressed

Dear Reviewers,

The authors posted point-by-point answers on 15 Nov, with additions updated on 16 Nov, including new baseline results and clarifications on the memory design and gating choices. Please read the response addressed to your review and post a short reply on whether your concerns are resolved. If issues remain, please say which ones and what would settle them; if they are resolved, a brief confirmation, and an updated rating where appropriate, would help the final assessment.

Thank you for your time. Area Chair

Dear Area Chair,

Thank you for your time and for handling our submission, including your reminder to the reviewers. We thank all reviewers for their constructive feedback.

Below, we summarize our responses following each reviewer’s original questions. “Addressed” indicates that the reviewer did not raise any further questions.

Reviewer rKVy

  • Q1. Difference from TrackVLA++’s gated memory: Clarified the maintained states, gate inputs, and branch-specific updates. (Addressed)
  • Q2(a). Motion information: The reviewer suggested additional experiments during the follow-up discussion, and we have provided the corresponding results in the comments.
  • Q2(b). Memory under occlusion: Clarified that closing write gates preserves memory reads and does not imply invisible-target extrapolation. (Addressed.)
  • Q3. OmTrackVLA results and baseline provenance: Added complete results, documented their sources, and corrected the Pareto figure and evaluation claims. (Addressed.)
  • Q4. Inconsistent ablation coverage: Extended component, encoder, and window-length experiments to STT/DT/AT. (Addressed.)
  • Q5. Zero-shot Gym-UnrealCV performance: Added evaluation across all three settings. (Addressed.)
  • Follow-up. Advantage over OmTrackVLA: Provided additional results in the comments demonstrating our performance advantages under matched-data and matched-backbone settings.

Reviewer jz7m

  • Q1. Baseline data, backbones, parameter counts, and matched comparisons: Added detailed accounting and matched-data/backbone OmTrackVLA experiments, plus ReferTrack retraining with remaining differences disclosed. (Addressed.)
  • Q2. Matched-hardware efficiency and module overhead: Added same-hardware measurements and separate latency, memory, and performance breakdowns for TMA and dual memory. (Addressed.)
  • Follow-up. Overall performance and larger-model evaluation: The reviewer acknowledged that most concerns were addressed, but retains reservations about the SR/TR gap. We have added experiments with larger-model alternatives.

Reviewer wgLM

  • Q1. Methodological novelty: Clarified the contributions of target-conditioned temporal adaptation and branch-specific memory updates beyond existing gated-memory designs. The reviewer retains reservations about novelty.
  • Q2. Controlled baseline comparisons: Added matched-backbone and matched-data OmTrackVLA experiments, as well as ReferTrack retraining on our tracking data, with remaining differences explicitly disclosed. The reviewer retains concerns about comparison fairness.
  • Q3. Release of data, code, and checkpoints: Specified the planned release of training, inference, and evaluation code, checkpoints, and data-generation and preprocessing tools. The reviewer requested further clarification on dataset availability; see our follow-up below.
  • Q4. Gating formulations and thresholds: Clarified which components are learned, manually specified, or calibrated from training data in Appendix A.4. (Addressed.)
  • Q5. Contributions of frozen backbones and simpler alternatives: Added encoder-removal ablations and comparisons with alternative video encoders and optical flow. (Addressed.)
  • Weakness 4. Limited real-world evaluation: Expanded testing from 42 trials across four environments to 104 trials per method across six environments. The reviewer acknowledged the additional evaluation but retains concerns about sequential trials.
  • Weakness 5. Incomplete efficiency comparisons: Added same-device system-level measurements and module-level latency and memory breakdowns, while disclosing remaining differences in timing scope and deployment settings. The reviewer retains concerns about runtime comparability.
  • Follow-up. Baseline comparisons: We further clarified that ReferTrack uses additional QA-based training stages and data absent from our pipeline, preventing fully matched training conditions. We therefore provided the closest feasible comparison by retraining its second stage on our tracking data and explicitly disclosed the remaining differences. Training-data availability: We clarified that our dataset can be reconstructed from EVT-Bench using the complete data-preparation pipeline we plan to release.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.