acceptodds
Under review as a conference paper at ICLR 2027

AlignTrack: Aligning Semantics, Geometry, and Actions for Embodied Visual Tracking

Abstract

Embodied visual tracking (EVT) requires agents to jointly understand target se- mantics, reason about spatial relations, and generate actions for continuous target following. While vision-language-action (VLA) models have shown strong poten- tial for EVT, their implicit action prediction leaves semantic, geometric, and action reasoning insufficiently coupled. To address this issue, we propose AlignTrack, a lightweight VLA that explicitly bridges these representations through a semantic- aligned depth encoder and an action-relevant understanding decoder. A two-stage strategy first aligns depth features with pretrained visual-semantic representations and then jointly supervises target-centric spatial understanding and action predic- tion. Extensive experiments on EVT-Benchmark show that AlignTrack with only a 0.6B backbone achieves performance comparable to the 7B TrackVLA, demon- strating the effectiveness of explicit spatial knowledge for lightweight embodied tracking. The code and trained models will be released publicly.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Tracking, VLA, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fiWLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AlignTrack, a lightweight vision-language-action framework for embodied visual tracking. It introduces a semantic-aligned depth encoder and an action-relevant understanding decoder, together with a two-stage training strategy that first aligns depth features with pretrained RGB representations and then jointly optimizes spatial understanding and action prediction. The proposed 0.6B model achieves competitive results against the 7B TrackVLA on EVT-Benchmark.

Critical strengths
  1. The paper addresses an important problem of explicitly connecting semantic understanding, geometric perception, and action generation in embodied tracking.

  2. The overall architecture is relatively lightweight and clearly structured, and the use of explicit target-state supervision is practically meaningful.

  3. The improvements on the AT setting are substantial, particularly in tracking rate and collision rate.

Critical weaknesses
  1. The main ablation only compares the full model with “w/o Depth” and “w/o Depth & Spatial Sup.” It therefore demonstrates the effect of adding depth, but does not establish whether the proposed semantic alignment pretraining is actually necessary. A direct comparison between aligned depth initialization and randomly initialized depth features is essential, especially since this control is mentioned in the appendix but no corresponding results are reported.

  2. AlignTrack uses additional metric-depth observations as input and relies on two pretrained visual encoders, DINOv3 and SigLIP, while the main efficiency argument primarily emphasizes the 0.6B language backbone. End-to-end parameter count, FLOPs, memory consumption, and inference latency should be reported for a fair efficiency comparison.
    The performance claims should be more carefully qualified. Although AlignTrack consistently improves tracking rate, it does not uniformly outperform TrackVLA. For example, on DT, AlignTrack obtains lower SR (53.70 vs. 57.6) and substantially higher CR (16.30 vs. 5.80). On STT, its SR is slightly lower and CR is also higher. The strongest overall advantage is mainly observed on AT.

  3. Bounding box, relative yaw, and visibility prediction are central to the proposed semantic–geometry–action coupling, yet the paper does not report the accuracy of these intermediate predictions. It is therefore unclear whether improved control truly results from better explicit spatial reasoning or simply from auxiliary supervision.

  4. Compared with “w/o Depth,” the full model improves collision rate but decreases both SR and TR on DT and AT. For example, on AT, SR/TR decrease from 66.64/81.63 to 64.93/79.68. This behavior deserves further analysis rather than being summarized only as a complementary benefit.

  5. Since the paper makes broader claims about lightweight embodied visual tracking and explicit spatial reasoning, evaluation in additional environments or under depth noise/domain shifts would provide stronger evidence of generalization.

Questions for authors

Please refer to Weaknesses.

Reviewer_V9hvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents AlignTrack, a lightweight vision-language-action model for embodied visual tracking that aims to jointly model semantic, geometric, and action information. The method adopts a two-stage training strategy, where depth features are first aligned with RGB representations through semantic alignment pretraining, followed by joint training for target-state and future-trajectory prediction. The model uses Qwen3-0.6B as the language backbone and incorporates RGB, depth, and temporal information. The experimental results show competitive performance across several settings. However, the core depth-alignment mechanism is not sufficiently validated, and some ablation results do not fully support the main claims of the paper. Therefore, I recommend clear reject.

Critical strengths
  1. Practically relevant problem. The paper studies the interaction among semantic understanding, geometric perception, and action prediction in embodied visual tracking. Compared with approaches that mainly focus on tracking performance, this setting considers how perceptual information supports subsequent action decisions, which is practically relevant.

  2. Relatively lightweight model design. The paper uses Qwen3-0.6B as the language backbone and achieves competitive results across several settings. This suggests that relatively small VLA models have potential for embodied visual tracking.

Critical weaknesses
  1. Figure 1 lacks a caption. Figure 1 does not provide a caption, making it difficult to understand the figure independently from the main text. Thus, this issue reduces the overall completeness and presentation quality of the manuscript.

  2. The available page budget is not utilized. The main paper is less than 7.5 pages, while the official limit is 9 pages, leaving considerable space for additional experiments and analysis. In particular, the alignment ablation, depth-branch analysis, and failure cases can be further discussed. Given that the current experimental validation remains incomplete, the available space could be used more effectively.

  3. The core depth-alignment mechanism is insufficiently validated. Depth-feature alignment is one of the central components of AlignTrack, but the current experiments do not sufficiently demonstrate that this mechanism itself provides a consistent performance gain. In particular, the RGB projector used to construct the alignment teacher comes from an RGB-spatial checkpoint, while the RGB projector in the joint-training stage is newly initialized and jointly optimized. Therefore, the alignment target during pretraining is not exactly the same as the final RGB representation, which raises concerns about the effectiveness of the proposed alignment strategy.

  4. A key alignment ablation is not reported. The appendix mentions a “Without aligned depth initialization” setting, but its result is not reported. This experiment is important for determining whether the observed improvement comes from the depth branch itself or from the proposed alignment pretraining. Without this result, the validation of the central methodological contribution remains incomplete.

  5. The depth branch does not consistently improve overall performance. According to Table 2, under DT, adding Depth decreases SR and TR by 2.82 and 1.96 percentage points, respectively, while improving CR by 4.61 points. Under AT, SR and TR also decrease by 1.71 and 1.95 points, while CR improves by only 1.10 points. These results suggest that the depth branch mainly improves collision-related behavior rather than consistently improving overall task performance. The paper should provide a clearer explanation of this trade-off.

  6. The analysis of the core mechanism is incomplete. The paper mainly evaluates the alignment mechanism through final task metrics, but provides limited direct analysis of the alignment quality itself. For example, the paper does not sufficiently analyze how the RGB and depth representations change before and after alignment. More controlled comparisons of different alignment strategies would make the main contribution more convincing.

Questions for authors
  1. Why does the alignment teacher use a different RGB projector from the one used during joint training? Does this mismatch affect the final depth-representation alignment?
  2. Why is the “Without aligned depth initialization” result not reported? How does the performance change when aligned depth initialization is removed?
  3. Why does adding Depth decrease SR/TR while improving CR under both DT and AT? Does this indicate that the depth branch mainly makes the policy more conservative?
  4. Can the authors provide more direct analysis of the alignment quality, such as measuring the similarity or distribution changes between RGB and depth representations before and after alignment?
  5. The paper emphasizes the lightweight nature of the model. Besides the size of the Qwen3 backbone, can the authors report the total parameter count, GPU memory usage, FLOPs, and inference latency?
Reviewer_9jK6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a 0.6B VLA tracking method for embodied visual tracking. The RGB frames are encoded by frozen DINOv3 and SigLIP. Qwen3-0.6B takes these features together with the language instruction to predict eight future waypoints. The paper adds two components. A small depth encoder is first pretrained to match the RGB tokens at the same image locations, and then initializes the depth branch for joint training. A spatial query predicts the target box, relative yaw and visibility as auxiliary supervision. On EVT-Bench, the paper reports results comparable to the 7B TrackVLA, with the largest gain in the AT setting.

Critical strengths

The auxiliary target-state supervision is simple and clearly helps in the ablation, especially on DT.

Critical weaknesses

  1. The paper's own ablation does not support its central component. In Table 2, the model without the depth branch has higher SR and TR on DT and AT. Depth only lowers the collision rate, yet the text calls it a complementary improvement. The alignment pretraining, which is the main idea of the paper, is never tested. Appendix E defines a variant with a randomly initialized depth branch, but its result is not reported. The design is also hard to justify. According to Appendix D, the depth tokens are aligned to an RGB projector from another checkpoint, but the joint run initializes a new RGB projector. The feature space that depth is aligned to therefore no longer exists during joint training.

  2. The comparison does not support the claim of being comparable to TrackVLA. AlignTrack uses depth input while TrackVLA uses only RGB. Even so, on DT it has a lower SR and almost three times the collision rate. TrackVLA++ and ReferTrack, which the paper discusses in Sec. 2.3, are missing from Table 1. The base model, which uses DINOv3 and SigLIP features with a Qwen3-0.6B backbone, matches the public OmTrackVLA. It should be cited and compared, but it is not mentioned. The lightweight claim is not backed by any speed measurement or real-world test. The evaluation covers only EVT-Bench, while TrackVLA additionally reports zero-shot results on Gym-UnrealCV. The manuscript also seems unfinished. The global batch size is missing, the strict success rate is mentioned but not reported, and the AT collision rate in Sec. 4.3 differs from Table 2.

Questions for authors
  1. What are the results with a randomly initialized depth branch? Why is the depth branch kept when removing it improves SR and TR on DT and AT?

  2. Why are the depth tokens aligned to an RGB projector that is not used in joint training?

  3. How does AlignTrack compare with OmTrackVLA, TrackVLA++ and ReferTrack? How does it compare with TrackVLA when both use only RGB?

  4. What is the inference speed on a robot platform?

  5. How does AlignTrack perform zero-shot on Gym-UnrealCV?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.