acceptodds
Under review as a conference paper at ICLR 2027

CST-WM: A Causally Structured World Model for Embodied Visual Tracking

Abstract

Embodied visual tracking requires a robot to choose actions that keep a moving target observable at a suitable distance, and to recover it after occlusion, out-of-view drift, or distractor crossings. We cast the task as planning over future target evidence with an action-conditioned world model. In logged tracking data, however, the behavior policy's actions are correlated with where the target is, so a generic predictor can learn a shortcut: it writes the current action directly into its prediction of target evidence, instead of letting the action affect that evidence only by moving the robot and changing what it observes. We call this failure causal hallucination; the resulting rollouts look plausible but rank candidate actions for the wrong reason. We propose CST-WM, a causally structured world model whose state is split into target-evidence, robot, and observation branches. Its transition removes the same-step edge from action to target evidence but keeps the path through robot motion and the resulting views, so candidate actions are still distinguished by their predicted ego-motion. With rollout-based model-predictive control, a single model handles both steady following and re-acquisition after target loss. On EVT-Bench and Habitat 3.0, covering standard tracking, target-loss recovery, and cross-dataset transfer, CST-WM improves following, distance-range control, safety, and re-acquisition over reactive trackers and world-model baselines, and removing the action mask causes the largest drop in re-acquisition among our ablations. Offline, CST-WM has lower multi-step rollout error, and its ranking of candidate actions agrees better with the simulator's. On a Unitree Go2 quadruped, CST-WM succeeds in 20 of 30 real-world trials under occlusion, distractor crossing, and fast motion, against 14 for TrackVLA.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, embodied visual tracking, causal structure, model-predictive control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HEVQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper proposes CST-WM, an action-conditioned world model for embodied visual tracking and target re-acquisition. Its state combines detector-derived target evidence, a visual latent, and the robot pose. The transition separates target-evidence, robot, and observation branches, using an attention mask to prevent the current action from directly entering the target-evidence branch. A detector-distilled evidence head scores imagined futures, and a CEM-based model-predictive controller selects actions.

Experiments on EVT-Bench and Habitat report improvements in following and recovery, supported by counterfactual action-ranking diagnostics and architectural ablations. Real-robot experiments provide additional evidence of applicability. My main concerns concern the positioning of the contribution, the justification of the representations and causal interpretation, and whether the recovery comparisons and computational settings sufficiently establish the method's practical advantage.

Critical strengths

S1. A meaningful problem with useful empirical evidence.

Maintaining visibility and recovering a moving target after occlusion are important problems for embodied tracking. Evaluating alternative actions through predicted future observations is a meaningful approach. The paper reports encouraging improvements across standard tracking, target-loss recovery, and cross-dataset transfer, together with a real-robot demonstration.

S2. The matched mask ablation provides evidence beyond overall task performance.

The comparison between the full model and the variant without masking is particularly informative. Removing the mask improves visibility prediction under logged actions, from 0.675 to 0.701 AUROC, but reduces counterfactual action-ranking correlation from 0.68 to 0.55 and closed-loop re-acquisition from 0.76 to 0.61. This distinction between predictive accuracy on logged actions and usefulness for planning is interesting. It supports the utility of the proposed constraint, although the precise explanation for that utility remains open.

Critical weaknesses

W1. The contribution and novelty require sharper positioning.

Predicting future observations or latent states, evaluating candidate actions, and selecting actions through model-predictive control are established ideas. Navigation World Models already uses action-conditioned visual prediction for planning, while WoMAP learns latent dynamics and detector-derived rewards for active perception. The latter is particularly relevant to the use of detection confidence, bounding-box size, and an evidence head for scoring imagined states.

The paper acknowledges these connections, but the distinction between established components, task-specific integration, and the proposed causal-routing contribution needs to be clearer. The principal methodological novelty appears to lie in the particular branch factorization and action mask. Its significance should therefore be established at that level, rather than inferred from the broader use of a world model for tracking and recovery.

W2. The representation design and its causal interpretation are insufficiently justified.

The state combines detector evidence H, a VAE latent Z, and robot pose x, but the paper does not sufficiently explain what information the learned target-evidence representation E captures, why the chosen compression and factorization are appropriate, or how they support the claimed separation of target dynamics, ego-motion, and observation changes. H provides confidence and apparent scale rather than target motion or identity, while Z is a distributed visual representation without an explicitly established physical interpretation. This matters particularly because the transition conditions on a single observation without a history or an explicit hidden-target belief.

The mask guarantees that E does not directly receive the current action. However, the robot branch retains an action embedding, and the evidence head reads observation-branch hidden tokens. Consequently, blocking the direct action-to-E connection does not by itself guarantee that action information influences predicted evidence exclusively through physically correct viewpoint changes. The reported gains support the architectural constraint, but do not fully distinguish the proposed causal explanation from a more general regularization or inductive-bias effect.

There is also a specific point requiring clarification: the action-independent E branch is supervised to predict whether the next relative robot–target distance is in range (Appendix A, Eq. 6), although that distance generally depends on the robot's current action. The compatibility of this supervision with the intended interpretation of E is unclear.

W3. Recovery comparisons and deployment-budget evidence do not yet fully establish the claimed advantage.

Table 2 compares recovery against TrackVLA and Adapted NWM. The matched NWM planner is a useful world-model baseline, but the table omits TrackVLA++, whose spatial reasoning and target memory directly address occlusion and target loss. OA-VAT also explicitly includes an occlusion-recovery planner. The paper explains that its Table 1 implementation uses the PID variant because the diffusion-planner checkpoint was unavailable. This is an understandable reproducibility limitation, but the comparison does not establish an advantage over OA-VAT's complete recovery approach.

The computational setting further limits interpretation. The default planner requires approximately 267 seconds per action on an RTX A6000, and simulation waits for planning to finish. The reported recovery times in simulation steps therefore exclude planning latency. The real robot uses a substantially smaller budget and achieves approximately 1.4 seconds per planning step on an offboard RTX 6000 Ada, which is valuable evidence and should be distinguished from the default cost.

However, the main mask ablations use the default budget. It remains unclear whether the same architectural advantage persists with the robot's shorter horizon and smaller candidate set, particularly when the target and robot continue moving during planning. A direct comparison under the deployment budget is important for connecting the central architectural claim to practical recovery.

Questions for authors

Q1. Contribution and novelty — corresponding to W1.

Please identify the principal methodological advance beyond NWM-style predictive planning and WoMAP-style detector-reward distillation. Which aspects are established components, which are adaptations to embodied tracking, and which constitute the new contribution? In particular, what new insight does the proposed factorization and action mask establish beyond a useful task-specific architectural constraint?

Q2. Representation and causal interpretation — corresponding to W2.

Please clarify what E is intended to represent, what supports that interpretation, and why the selected combination and compression of H, Z, and x is appropriate for tracking and recovery. How is the action-independent E branch's next-distance supervision in Eq. 6 consistent with that interpretation?

Please also distinguish the dependencies guaranteed by the mask from the physical or causal properties supported empirically. Since action information can still reach the evidence head through the robot and observation branches, what evidence distinguishes mitigation of the proposed action shortcut from a general regularization effect? A focused control on a representative subset, such as a comparably restrictive alternative mask or an evaluation that changes the logging-policy correlation, would help resolve this question.

Q3. Recovery comparisons and computational budget — corresponding to W3.

Please explain the selection of recovery baselines in Table 2, especially the omission of TrackVLA++ and the scope of conclusions possible without OA-VAT's complete recovery planner. If the stronger recovery baselines cannot be evaluated, please clarify the resulting limits of the comparative claims.

Can the full model and the existing variant without masking be compared under the robot deployment budget on a representative subset of the existing recovery episodes? Please report recovery performance and complete planning latency, and explain how the model's transition interval relates to the 1.4-second planning cycle, continued execution of the previous action, and observation staleness during planning.

6 comments
AuthorsOpenReview ↗

Response to Reviewer HEVQ Q1: Contribution and Novelty (W1)

Thank you for the review.

We separate the contribution into three levels. The methodological advance is the action-routing constraint, together with the evidence that accuracy on logged futures can rank world models in the wrong order for planning.

Established components. An action-conditioned diffusion world model planned with CEM (NWM), and a reward head distilled from a detector that scores latent states without decoding (WoMAP). We use both and cite both where they enter (Sections 2 and 3.2.2).

Adaptation to tracking. The objective of Eq. 5 scores predicted visibility and apparent scale; it replaces NWM's goal image, which tracking does not have. The evidence head is trained jointly with the transition. Joint training reaches re-acquisition 0.76, against 0.43 for the same head on a frozen transition (Table 10).

New contribution. Our method introduces three new elements:

(1) A confound in logged tracking data. The behavior policy chooses its action according to where the target is, so a transition can predict next-step evidence from the action without modeling what the action does. In our logs the correlation between the action and the target bearing is 0.55, and the mask's advantage shrinks as this correlation is removed (Table R7, HEVQ Q2).

(2) A transition that removes the one edge this confound uses, aℓ→Eℓ+1a_{\ell} \to E_{\ell+1}, and keeps the action's effect through ego-motion. A placebo mask of the same size does not recover its gain (HEVQ Q2).

(3) A paired-intervention protocol that tests the difference where planning needs it, on actions the behavior policy did not take. It uses 500 saved states, each with 32 candidate sequences executed in the simulator and imagined by every model (Section 4.4). It locates the mask's gain on the candidates far from the logged action (HEVQ Q2).

Insight beyond a task-specific constraint. A world model that is more accurate under the logging policy is not necessarily better for planning. The unmasked model predicts logged futures better (visibility AUROC 0.701 against 0.675, Table 4), ranks counterfactual candidates worse (ρ 0.55 against 0.68) and recovers worse in closed loop (0.61 against 0.76). The adapted NWM, with 6.6× our parameters, does not rank better (ρ 0.47). We expect the same confound wherever a world model is trained on goal-directed logs, because an expert chooses its action based on the task-relevant state. The remedy we test is to compute the task-relevant representation before the action is read. The revised PDF states these three levels in the last paragraph of the introduction and in the related work.

AuthorsOpenReview ↗

Response to Reviewer HEVQ Q2: Representation and Causal lInterpretation (W2)

What E represents, and why H, Z and x. Following your question on what E is intended to represent, we define Eℓ+1E_{\ell+1} as the target-related content of the current state, computed before the action is seen: whether the target is in view, where, and at what distance. To verify this reading, we conduct linear probes on the frozen E (Table R6). Target visibility is decoded with accuracy 0.92, in-range distance with 0.85, and the target's horizontal image position with R² 0.63. When the action is swapped at a fixed state, E does not follow it, while the unmasked branch does (R² 0.01 against 0.45). With the humanoid held still and the robot moving, E stays stable (0.98 against 0.61 for the adapted NWM, Table 3).

Table R6 (Table 18 in the revised PDF) Linear probes on the frozen Eℓ+1E_{\ell+1} (held-out Habitat 3.0 transitions).

Probe targetFullW/o masking
Target visible at ℓ+1 (accuracy)0.920.92
Distance in following range at ℓ+1 (accuracy)0.850.88
Target horizontal image position (R²)0.630.65
Swapped action aℓa_{\ell} at a fixed state (R²)0.010.45

The state components each cover what the others cannot. H holds the two quantities the task is scored on, target in view and distance in range. Z holds the rest of the scene, the occluders and free space that determine what the camera sees after it moves. The pose x lets ego-motion follow kinematics instead of being learned. The scalar compression of H enters only as the transition's input. Replacing the trained signal with the raw detector score lowers DRS from 0.70 to 0.60 and re-acquisition from 0.76 to 0.66 (Table 10). Target identity and a hidden-target belief are outside our contribution, which concerns how the action reaches the target evidence; Appendix E states this boundary. Within it, recovery works by replanning at every step from a fresh detector reading. Only the last executed action carries over, through the smoothness limit of the admissible set.

Next-distance supervision in Eq. 6. Following your question on how this supervision is consistent with an action-free E, the answer is that gdistg_{\text{dist}} learns the label averaged over the behavior policy's actions, P(dℓ+1 in range∣Hℓ,Zℓ,xℓ)P(d_{\ell+1}\ \text{in range} \mid H_{\ell}, Z_{\ell}, x_{\ell}). E can represent this average, and the mask's benefit does not rest on this loss. The label differs from the same label at step ℓ on 2.1% of training transitions. The loss cannot introduce an action dependence, because no gradient path from aℓa_{\ell} to E exists. Without LdistL_{\text{dist}}, re-acquisition is 0.72; without the mask, 0.61 (Table 8). Details are in RrzZ Q1.

Guaranteed dependencies and evidence for shortcut mitigation. Following your question on which dependencies the mask guarantees and which are shown empirically, the split is as follows. Guaranteed: Eℓ+1E_{\ell+1} does not depend on aℓa_{\ell} (Eq. 2). Not guaranteed: that the head's reading H^ℓ+1\hat H_{\ell+1} depends on the action only through a physically correct view change. Action information reaches the observation tokens through the robot branch, as it must for the view to move. To separate shortcut mitigation from a general regularization effect, we conduct both controls you suggest and a third.

(1) Changing the correlation with the logging policy at training time (Table R7). We retrain the masked and the unmasked transition on Habitat 3.0 data re-collected with a fraction ε of the actions replaced by random admissible actions. As the correlation between logged action and target bearing falls from 0.55 to 0.05, the mask's advantage falls from 0.13 to 0.03 in Plan. ρ and from 0.15 to 0.02 in re-acquisition.

(2) A comparably restrictive alternative mask (Table R2a in the common response). A placebo mask removes the same number of attention edges from the target-evidence branch and keeps the action edge. It reaches re-acquisition 0.63 and Plan. ρ 0.56, slightly above no mask (0.61, 0.55) and below the action mask (0.76, 0.68).

(3) Changing the correlation at test time (Table R2c). The mask adds 0.02 to Plan. ρ on the 16 candidates nearest to the logged action sequence and 0.19 on the farthest 16.

Table R7 (Table 23 in the revised PDF) The mask's advantage as the logged action is decorrelated from the target. ε: fraction of logged actions replaced by random admissible ones.

ε = 0 (paper)ε = 0.5ε = 1.0
Correlation between action and target bearing0.550.310.05
Plan. ρ↑, full0.680.660.62
Plan. ρ↑, w/o masking0.550.600.59
Plan. ρ↑, gap0.130.060.03
Re-acq.↑, full0.760.730.65
Re-acq.↑, w/o masking0.610.660.63
Re-acq.↑, gap0.150.070.02
AuthorsOpenReview ↗

Response to Reviewer HEVQ Q3: Recovery Comparisons and Computational Budget (W3)

Recovery baselines in Table 2. Following your question on why Table 2 omits TrackVLA++ and what can be concluded without OA-VAT's complete recovery planner, we answer in two parts. The recovery claim holds against TrackVLA, TrackVLA++ and the adapted NWM. We make no claim over OA-VAT's planner. For the first part, we conduct the recovery experiment with our reimplementation of TrackVLA++ on the four target-loss conditions (Table R5).

TrackVLA++ has released neither code nor weights (arXiv 2510.07134; its project page links none), so its row in Table 1 has two sources, and the submission did not distinguish them.

  • EVT-Bench columns: reported numbers. SR 86.0, TR 81.0 and CR 2.10 are the single-view results of Table I in the TrackVLA++ paper.
  • Transfer columns on Habitat 3.0: our reimplementation. We built the model from the description in its paper. We started from the architecture and released code of TrackVLA and added the two TrackVLA++ modules, Polar-CoT spatial reasoning and the Target Identification Memory. Its training data follows the paper where we could obtain it; about 20% of that data could not be obtained or reproduced.
  • Fidelity of the reimplementation. In our evaluation on EVT-Bench STT, it reaches SR 84.3, TR 80.2 and CR 2.26, against the reported 86.0, 81.0 and 2.10.

Table R5 (Table 2 in the revised PDF) Table 2 with TrackVLA++ added (Habitat 3.0 target-loss protocol). TTR is averaged over successfully re-acquired episodes only. Post-F and Rec-ES are in Table 2.

Re-acq.↑Short occlusionLong occlusionOut-of-view driftDistractor crossing
TrackVLA0.710.480.520.43
TrackVLA++ (our reimpl.)0.730.570.570.51
Adapted NWM0.750.540.570.49
Ours0.840.690.730.65
TTR↓ (steps)Short occlusionLong occlusionOut-of-view driftDistractor crossing
TrackVLA8.914.612.815.2
TrackVLA++ (our reimpl.)8.3912.7311.8313.87
Adapted NWM8.113.211.714.0
Ours6.49.88.710.6

TrackVLA++ improves on TrackVLA in every condition, most under long occlusion (0.57 against 0.48), where its target memory applies. Across short occlusion, long occlusion, out-of-view drift and distractor crossing, CST-WM stays ahead in re-acquisition by 0.11, 0.12, 0.16 and 0.14. Its mean TTR over each method's successful episodes is lower by 1.99, 2.93, 3.13 and 3.27 steps. The revised PDF marks reported and reimplemented numbers with separate symbols in Table 1 and describes the reimplementation in Appendix C.

On OA-VAT: the checkpoint of its diffusion planner is not released, so we ran its released pipeline with PID control (Appendix C). OA-VAT appears only in Table 1, and the revised PDF states this scope.

Full model against the unmasked variant at the deployment budget. The architectural advantage does not depend on the long horizon or the large candidate set, and it holds when the environment keeps moving during planning. Following your request, we conduct the comparison at the robot budget on 200 recovery episodes, 50 per condition (Table R1 in the common response). At the robot budget with the simulator paused, re-acquisition is 0.68 with the mask and 0.47 without. With the simulator running through each planning cycle, it is 0.55 and 0.42. TTR, which now counts the steps that pass during planning, is 11.67 and 15.11. The advantage is 0.21 with the simulator paused and 0.13 with it running, against 0.22 at the default budget.

Complete planning latency and timing. One planning step at the robot budget takes 1.4 s on the RTX 6000 Ada from the received observation to the issued action: 80 sequential denoising passes over 24 candidates, 8 evidence-head reads, one detector call and one encoder call (Table 20). One model transition spans one control step, Δt = 0.5 s, so the simulation rounds the cycle up to 3 control steps (1.5 s). The horizon T = 4 covers 2.0 s. During a cycle the robot repeats the action selected by the previous cycle. The plan is computed from the observation taken at the start of the cycle, so that observation is 1.5 s old when the first action is issued. This delay accounts for the full model's drop of 0.13 relative to the paused simulator (0.05 for the unmasked model).

Reviewer_2pSYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

CST-WM studies how action-conditioned world models can support embodied tracking and target recovery without exploiting misleading correlations in logged data. It separates target evidence, robot state, and observations, restricting the direct influence of actions on the target-evidence branch while retaining their effect through robot motion. The resulting model predicts future target observability to guide planning. Simulation benchmarks, action-ranking diagnostics, ablations, and real-robot trials report improvements in following and re-acquisition and examine the role of the proposed structural constraint.

Critical strengths
  • The matched masking ablation directly supports the architectural contribution: removing the mask reduces re-acquisition from 0.76 to 0.61 and candidate-ranking correlation from 0.68 to 0.55 (Table 6). The paired simulator interventions provide more relevant evidence for planning than reconstruction accuracy alone.
  • Results extend across tracking and transfer (Table 1), with real-robot success of 20/30 versus 14/30 (Table 5). Appendix C appropriately reports uncertainty and discloses additional real-frame training for deployment.
Critical weaknesses
  • The architecture guarantees action independence of the internal target branch, but does not clearly guarantee the claimed exclusively physical influence on predicted evidence. The robot representation explicitly contains an action embedding, and the evidence head reads observation tokens rather than the generated image (Section 3.2.3, Eq. 2). An action shortcut through these tokens remains possible. Appendix D.1 acknowledges that the zero-Jacobian check verifies the mask; improved rankings support an inductive bias, but do not establish that shortcuts have been excluded.
  • The main simulation results use a planner taking 267 seconds per action on an A6000 while simulation waits (Section 4.6, Table 16). Deployment uses a substantially smaller budget at 1.4 seconds per action. The real-robot results are encouraging but limited, leaving the main performance advantage under practical timing constraints uncertain.
Questions for authors
  1. Clarification: What constrains the action embedding in the robot branch from influencing the evidence head independently of physically correct view changes? How should Eq. 3's factorization be interpreted when the head reads shared tokens?
  2. Can you report the matched masking comparison at the robot planning budget with the environment advancing during computation, to establish whether the central recovery gain survives deployment timing?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2pSY Q1: Action Pathway and the Interpretation of Eq. 3

Thank you for the review.

Action pathway. Following your concern about the action embedding and a head that reads observation tokens, not the generated image, we agree: no architectural constraint rules out every shortcut from the robot branch to the evidence head. The guarantee is Eq. 2: a zero Jacobian on the internal target representation Eℓ+1E_{\ell+1}, not on the head's output H^ℓ+1\hat H_{\ell+1}. H^ℓ+1\hat H_{\ell+1} must depend on aℓa_{\ell} within the same step, or candidates could not be told apart. The revised PDF rewords the sentences that claimed more than the guarantee (listed in the common response). In the trained model, this dependence runs through the change of view, and the mask's benefit is not confined to the token-based read-out. We conduct four experiments to test this (Tables R2, R7 and R8).

(1) Distance from the logged action. The mask adds 0.02 to Plan. ρ on the candidates nearest to the behavior policy's action and 0.19 on the farthest; the full model ranks both halves equally well (0.64, 0.65). A model that had learned "what the behavior policy sees after this action" would rank well near that policy and badly away from it, as the unmasked model does.

(2) Strength of the confound. When both models are retrained on logs with randomized actions, the mask's advantage in Plan. ρ falls from 0.13 to 0.06 with half of the actions randomized and to 0.03 with all of them.

(3) Read-outs of the generated latent. With a detector on the decoded image and with a separate latent head, the mask's advantage in Plan. ρ is 0.11 and 0.09, against 0.13 with the evidence head. The two models already differ in the images they generate. These read-outs rank less well overall (0.58 and 0.54 against 0.68).

(4) Intervention at a fixed predicted view (Table R8). We vary the action given to the robot branch and re-evaluate the observation tokens and the evidence read-out. E, the final predicted latent Ẑ and its decoded view stay unchanged. The full model's predicted confidence changes by 0.0007, against 0.034 when the swap also changes the view. For the unmasked model the two numbers are 0.0016 and 0.025. When the final pass is given the same mean action for every candidate, the full model's Plan. ρ falls from 0.68 to 0.64 and the unmasked model's from 0.55 to 0.47.

Table R8 (Table 24 in the revised PDF) Intervening on the action at a fixed predicted view (500 states × 32 candidates). All conditions, including the view-changing control, are measured under the same paired-intervention protocol.

Read-out conditionFullW/o masking
Change of predicted confidence when the action is swapped throughout (the view changes)0.0340.025
Change when the action is swapped only in the final denoising pass (predicted view held fixed)0.00070.0016
Plan. ρ, normal read-out0.680.55
Plan. ρ, final pass run with a candidate-independent mean action (view held fixed)0.640.47

The pose-only robot branch (Table R2a) gives similar results to the paper's model. The next pose still carries the action through the kinematics, so that row shows the embedding is redundant; it is not a test of the shortcut.

Interpretation of Eq. 3. Following your question on how to read the factorization when the head reads shared tokens, we now state its scope. Eq. 3 is exact for a read-out that takes the generated latent as input, such as the detector on the decoded view. For our token-based head it is an approximation. The implemented paths are a → R → hᴼ → Ĥ and hᴼ → Ẑ: evidence and latent prediction are separate outputs of the observation tokens. The evidence head reads the final-layer tokens from which Z^ℓ+1\hat Z_{\ell+1} is produced by a linear projection, so it is not a function of Z^ℓ+1\hat Z_{\ell+1}. We chose it for two reasons. It reads 128 imagined views in 0.2 ms, where decoding and detecting takes 49 ms per view. It also reaches re-acquisition 0.76 in closed loop, against 0.46 for decode-and-detect (Table 10). The revised PDF states this next to Eq. 3.

AuthorsOpenReview ↗

Response to Reviewer 2pSY Q2: Matched Masking Comparison Under Deployment Timing

The mask's recovery advantage depends neither on the paused simulator nor on the large default budget. Following your request, we conduct the matched comparison at the robot budget (T=4, N=24, K=8, M=2, 10 sampling steps; Table R1 in the common response). The simulator advances three control steps (1.5 s) during each planning cycle, and the robot repeats its previous action meanwhile.

The full model re-acquires the target in 0.55 of episodes and the unmasked model in 0.42. The gap is 0.13, against 0.22 at the default budget with the simulator paused. The smaller budget costs the full and unmasked models 0.05 and 0.04, respectively; the delay costs them a further 0.13 and 0.05. The full model issues 3.87 plans between loss and re-acquisition. A controller that holds the last action re-acquires in 0.29 of episodes, and TrackVLA, a reactive policy that acts at every control step, in 0.44.

Thank you for the thorough rebuttal and the additional experiments. They address both of my original questions well, and I have raised the presentation and soundness scores accordingly.

After reading the other reviewers' comments, the authors' responses to them, and the paper again, I still have some reservations about significance, so I keep my overall rating for now:

As Reviewer HEVQ noted, the main novelty is the action mask on top of established components. The new decorrelation experiment in the response to HEVQ (Table R7) suggests that the mask's benefit depends mainly on the correlation in the logged data. Following Reviewer RrzZ's point on target identity, the response clarifies that the model keeps no memory of a hidden target. This seems to limit recovery to cases where the target reappears within the planning horizon.

I appreciate the authors' careful and honest clarifications, and I think this is a solid study of a useful design choice.

Reviewer_RrzZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates embodied visual tracking as planning over predicted target visibility and apparent scale. The world model separates target-evidence, robot, and observation branches, blocks direct current-action input to the target-evidence branch, and uses a detector-distilled evidence head to score imagined futures for CEM planning. Evaluation includes standard tracking, target-loss recovery, transfer, simulator interventions, and preliminary real-robot trials.

Critical strengths
  1. The paper addresses an important distinction between prediction under logged actions and prediction for planning over alternative actions. The paired simulator protocol uses 500 saved states and the same 32 candidate sequences for each model, directly testing action ranking rather than only logged-trajectory prediction (Section 4.4 and Appendix C). There is meaningful same-backbone evidence for the structural mask. Removing the mask reduces re-acquisition from 0.76 to 0.61 and planning-value Spearman correlation from 0.68 to 0.55 (Table 6). The ablations of factorization, update order, evidence readout, and auxiliary loss make the empirical case more informative.

  2. The evaluation includes transfer and real hardware, and the manuscript reports important boundaries explicitly. The real-robot success counts are 20/30 versus 14/30 for TrackVLA. The authors also disclose planning cost, the paused simulation protocol, the reduced deployment budget, additional evidence-head adaptation, and the absence of persistent target identity.

Critical weaknesses
  1. The causal interpretation is stronger than the architectural constraint alone establishes. Although the target-evidence branch excludes the current action, the robot representation includes both the next pose and a raw action embedding, and the evidence head reads observation-branch hidden tokens. Thus, zero action dependence in the target-evidence branch does not by itself exclude an action shortcut in the eventual planning score. The next-distance auxiliary label on the action-invariant branch also needs explanation because relative next-step distance depends on robot motion (Section 3.2.3 and Appendix A). Prediction at visibility changes remains weak. Appendix D.2 reports AUROC 0.467 on steps whose visibility differs from the initial frame, versus 0.530 for detection on decoded views. This needs to be reconciled with the recovery results, particularly because the model uses only the current view and lacks a persistent target identity.

  2. The main simulation results do not incorporate planning latency. A default planning step takes 267 s on an A6000, while the reduced robot budget takes 1.4 s on an RTX 6000 Ada; the simulator waits for planning (Table 16). This makes the main results evidence about decision quality with a paused environment. Evaluation at the actual control rate would clarify how much of the advantage persists for a moving target.

My recommendation is borderline, leaning toward acceptance. The problem is compelling, and the intervention-based ranking evaluation and same-backbone masking ablation provide a credible empirical contribution. However, the evidence currently supports an effective structural inductive bias more clearly than the stronger physical causal interpretation, and the latency gap materially affects deployment relevance. Clarifying the action pathway and demonstrating that the gains survive a realistic control budget would move my assessment toward acceptance.

Questions for authors
  1. Does the ranking/recovery advantage persist when the robot branch carries only the next pose, without the raw action embedding? How is the action-invariant target branch reconciled with next-distance supervision?
  2. Can the authors report tracking and recovery with the deployment budget and measured observation-to-action delay in a continuously running simulator?
  3. How should the 0.467 AUROC on visibility-changing steps be interpreted? Can disappearance and reappearance be evaluated separately, and can the planning benefit be decomposed into visibility, scale, and motion-penalty terms?
4 comments
AuthorsOpenReview ↗

Response to Reviewer RrzZ Q1: Pose-only Robot Branch and Next-distance Supervision

Thank you for the review.

Action pathway and the pose-only robot branch. Following your concern about the raw action embedding and the token-based head, we agree: the architecture alone does not exclude a shortcut in the planning score. The guarantee is Eq. 2: a zero Jacobian on Eℓ+1E_{\ell+1}, the internal target representation, not on the head's output H^ℓ+1\hat H_{\ell+1}. H^ℓ+1\hat H_{\ell+1} must depend on aℓa_{\ell} within the same step, or candidate actions could not be told apart. Nothing in the architecture forces that dependence to run through the change of view. The revised PDF rewords the sentences that claimed more (listed in the common response). In the trained model it does. To test this, we conduct the pose-only experiment you ask for and four further controls.

(1) Pose-only robot branch. Retrained with Rℓ+1=[xℓ+1]R_{\ell+1} = [x_{\ell+1}], the model reaches re-acquisition 0.75 and Plan. ρ 0.67, against 0.76 and 0.68 with the embedding. Without the mask it drops to 0.62 and 0.56 (Table R2a). The mask's advantage in ranking and recovery persists. Because xℓ+1=fx(xℓ,aℓ)x_{\ell+1} = f_{x}(x_{\ell}, a_{\ell}) still carries the action, this result shows that the embedding is redundant; it does not by itself exclude a shortcut.

(2) Evidence read from the generated latent. With GroundingDINO on the decoded view, Plan. ρ is 0.58 with the mask and 0.47 without; with a separately trained latent head, 0.54 and 0.45 (Table R2b). The gaps of 0.11 and 0.09 are close to the 0.13 measured with the evidence head, so the two models already differ in the views they predict.

(3) Candidates far from the behavior policy. On the 16 candidates nearest to the logged action sequence the mask adds 0.02 to Plan. ρ; on the farthest 16 it adds 0.19 (Table R2c). The unmasked model ranks well where the behavior policy has been and poorly elsewhere, while the full model ranks both halves equally well (0.64, 0.65).

(4) Weaker training correlation. We retrain both models on logs in which a fraction of the actions is replaced by random admissible ones. The mask's advantage in Plan. ρ falls from 0.13 to 0.06 with half of the actions randomized and to 0.03 with all of them (Table R7, HEVQ Q2).

(5) Fixed predicted view. We swap the action only in the pass the head reads, so the predicted view stays fixed. The predicted confidence changes by 0.0007, against 0.034 when the swap also changes the view (Table R8, 2pSY Q1).

Next-distance supervision on the action-free branch. Following your concern that the next-step distance depends on robot motion yet supervises the action-free branch, we show that the label the auxiliary head learns is close to a function of the current state, and that the mask's effect does not depend on this loss. Because gdistg_{\text{dist}} reads only E, it learns the expectation of the label over the actions the behavior policy takes from that state, P(dℓ+1 in range∣Hℓ,Zℓ,xℓ)P(d_{\ell+1}\ \text{in range} \mid H_{\ell}, Z_{\ell}, x_{\ell}). We measure how often the label changes within one step. It differs from the same label at step ℓ on 2.1% of training transitions, and some admissible action could flip it on 3.9%. The robot moves forward and turns, without lateral translation, and its own motion contributes at most vmax⁡Δt=0.3v_{\max}\Delta t = 0.3 m per control step. The target's own motion can add to this. On those transitions the action-dependent part is noise that E cannot fit. The gradient cannot create an action dependence either, because no path from aℓa_{\ell} to E exists.

The loss is there so that E carries metric distance; apparent scale gives distance only up to body size. It raises the rank correlation between the signal and true distance across humanoids from 0.71 to 0.83 (Table 19). Without LdistL_{\text{dist}} the model keeps re-acquisition 0.72 and Plan. ρ 0.64. Without the mask it has 0.61 and 0.55 (Table 8). The revised PDF describes the label as a one-step-ahead reading of the current geometry and reports both percentages.

AuthorsOpenReview ↗

Response to Reviewer RrzZ Q2: Deployment Budget and Measured Delay in a Running Simulator

Following your concern that the main simulation results exclude planning latency, we agree: the main tables measure decision quality with a paused environment. The mask's recovery advantage also holds at the control rate, although the smaller budget and the delay both reduce absolute performance. To verify this, we conduct the experiment you describe (Table R1 in the common response): the robot budget, a simulator that keeps running during planning, and a 1.5 s simulated delay. The delay is the measured 1.4 s from observation to action, rounded up.

The full model reaches re-acquisition 0.55, TTR 11.67 steps, F 0.49 and DRS 0.64; without the mask the numbers are 0.42, 15.11, 0.39 and 0.55. Against the paused simulator at the default budget, the full model loses 0.18 in re-acquisition: 0.05 from the smaller budget and 0.13 from the delay. The mask's advantage narrows from 0.22 to 0.13. The full model issues 3.87 plans between loss and re-acquisition. A controller that holds the last action re-acquires in 0.29 of episodes, and TrackVLA, which acts at every control step, in 0.44. The revised PDF states that the main tables are measured with a paused simulator and adds Table R1 to Section 4.6 as the control-rate result.

AuthorsOpenReview ↗

Response to Reviewer RrzZ Q3: Visibility Changes and Decomposition of the Planning Benefit

Visibility prediction: disappearance and reappearance. Following your concern about the steps whose visibility differs from the initial frame, we keep the pooled number in the revised PDF as a limitation: the head scores 0.467, below detection on decoded views (0.530). Recovery rests on a narrower ability: telling apart the candidates that bring a hidden target back. The pooled score understates it because it mixes two kinds of step with opposite labels. Disappearance steps start visible and are labeled invisible; reappearance steps start invisible and are labeled visible. The model's input includes the current evidence, so its prediction starts high on the first kind and low on the second. Once the two are pooled, the positives begin with lower scores than the negatives. We conduct the two evaluations you ask for to separate the two kinds (Table R3).

Table R3 (Table 16 in the revised PDF) Visibility prediction split by the starting state. Top: logged-action rollouts of Appendix D.2 (192 starts, 8 steps). Bottom: paired interventions of Section 4.4, states that start with the target hidden; AUROC over the 32 candidates of each state, averaged over states.

Read-outStart visible: disappearance↑Start hidden: reappearance↑
Copy current evidence0.500.50
Detect on decoded view0.610.56
Separate latent head0.620.55
Head, u=0 (ours)0.630.64
Head, w/o masking0.690.66
Under interventions (which candidates bring the target back)Reappearance↑
Adapted NWM0.54
W/o masking0.59
Full0.76

Split by starting state. Among rollouts that start with the target visible, the head predicts which steps lose it with AUROC 0.63; among rollouts that start with it hidden, it predicts which steps regain it with AUROC 0.64. Copying the current evidence scores 0.50 on both by construction, and detection on the decoded view scores 0.61 and 0.56.

Under interventions. For each hidden-target state, we test whether the head's predicted visibility picks out the candidates that bring the target back into view in the simulator. A candidate is positive if the target reappears at any step within the horizon. AUROC over the 32 candidates is 0.76, against 0.59 without the mask and 0.54 for the adapted NWM. Under logged actions the unmasked model is as good or slightly better (0.66 against 0.64), in line with Table 4; under interventions it is worse, in line with Table 8.

Persistent target identity and a hidden-target belief are outside our contribution, which concerns how the action reaches the target evidence; Appendix E states this boundary. Within it, a single-view model recovers targets as follows. Each candidate moves the camera differently, the model predicts the view each one leads to, and the planner replans from a fresh detector reading at every step. Only the last executed action carries over: the smoothness limit of AvalidA_{\text{valid}} keeps the search near the turn the robot was already making. Section 3.3 of the submission said that the target-evidence branch "carries the last evidence"; that wording was wrong, and the revised PDF replaces it with this description.

Decomposition into visibility, scale and motion penalties. Following your request to split the planning benefit into its terms, we find that each term has its own role. Two experiments show this: we score the paired-intervention candidates with each term of Eq. 5 alone, and we remove each term in closed loop (Table R4).

Table R4 (Table 17 in the revised PDF) Contribution of the reward terms. Top: Plan. ρ when candidates are scored by a subset of the terms (500 states × 32 candidates). Bottom: closed loop on Habitat 3.0 with one term removed (full model).

Terms used for scoringFullW/o maskingGap
Visibility only0.540.400.14
Scale only0.500.420.08
Visibility + scale0.590.460.13
All terms (paper)0.680.550.13
Closed loop, full modelF↑DRS↑Re-acq.↑CR↓
All terms (paper)0.530.700.760.27
W/o visibility term0.480.640.510.30
W/o scale term0.470.530.720.33
W/o validity and safety penalties0.510.680.730.37

Removing the visibility term lowers re-acquisition from 0.76 to 0.51. Removing the scale term lowers DRS from 0.70 to 0.53 and leaves re-acquisition at 0.72. Removing the penalties raises CR from 0.27 to 0.37. The mask's gain is concentrated in the visibility term (0.14 in ρ, against 0.08 for scale).

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer RrzZ

Dear Reviewer RrzZ,

Thank you again for your review. We have responded to your three questions with additional experiments and clarifications:

  • Q1: Pose-only robot branch and next-distance supervision (Table R2).
  • Q2: Deployment-budget evaluation with the simulator advancing during planning (Table R1).
  • Q3: Visibility prediction for disappearance and reappearance, and the contributions of individual planning reward terms (Tables R3 and R4).

We would appreciate hearing whether our responses have addressed your concerns, or whether there are any remaining points you would like us to clarify. We would be happy to continue the discussion.

Best regards,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response: New Experiments on the Two Shared Questions

We thank the reviewers for their careful reading. Two questions are shared across the reviews, and each is answered by one new table. All rows use submission checkpoints unless marked as retrained, and paper table numbers refer to the revised manuscript.

1. The mask's recovery advantage holds at the deployment budget while the environment keeps moving during planning. We conduct the target-loss experiment at the robot budget in a simulator that keeps running through each planning cycle (Table R1). The full model re-acquires the target in 0.55 of episodes, against 0.42 without masking. The mask's advantage is 0.22 at the default budget, 0.21 at the robot budget and 0.13 with the simulator running. TrackVLA is not trained on Habitat 3.0, so its like-for-like row is CST-WM trained on EVT-Bench only: 0.52 against 0.44. The analysis is in RrzZ Q2 and 2pSY Q2, and the complete latency and timing relations are in HEVQ Q3.

Table R1 (Table 7 in the revised PDF) Robot planning budget with a continuously running simulator (Habitat 3.0 target-loss protocol, 200 episodes, 50 per condition; F and DRS on the standard-tracking subset). Running: the simulator advances 3 control steps (1.5 s, the measured 1.4 s rounded up) in each planning cycle while the robot repeats its previous action. Plans: planning decisions issued between target loss and re-acquisition. TTR and Plans are averaged over the same successfully re-acquired episodes for each model; TTR counts the steps elapsed during planning. The default rows are re-measured on these episodes.

ModelBudget (T,N,K,M,S)SimulatorCycle (s)Re-acq.↑TTR↓ (steps)PlansF↑DRS↑
Fulldefault (10,128,16,4,20)pausedn/a0.738.888.880.530.70
W/o maskingdefaultpausedn/a0.5112.4212.420.470.63
Fullrobot (4,24,8,2,10)pausedn/a0.689.839.830.500.65
W/o maskingrobotpausedn/a0.4713.1713.170.440.60
Fullrobotrunning1.400.5511.673.870.490.64
W/o maskingrobotrunning1.400.4215.115.070.390.55
Full, trained on EVT-Bench onlyrobotrunning1.400.5212.424.080.430.60
Hold last action (no replanning)n/arunningn/a0.2919.430.000.200.42
TrackVLA (not trained on Habitat 3.0)one pass per steprunning0.300.4412.83n/a0.310.47

2. The mask guarantees only that E does not depend on the current action. Whether the planning score follows the action through the predicted view is an empirical question, and we test it. The revised manuscript narrows its wording to the guarantee, ∂Eℓ+1/∂aℓ=0\partial E_{\ell+1}/\partial a_{\ell} = 0. We conduct three controls on the paired interventions (Table R2): (a) a pose-only robot branch and a placebo mask, (b) read-outs of the generated latent, and (c) candidates split by distance from the logged action. The placebo mask does not recover the gain. The gain persists under latent read-outs (0.11 and 0.09, against 0.13) and grows away from the logged action (0.02 against 0.19). The discussion is in RrzZ Q1, 2pSY Q1 and HEVQ Q2. Two further experiments support it: decorrelated training (Table R7, HEVQ Q2) and the fixed-view intervention (Table R8, 2pSY Q1).

Table R2 (Table 5 in the revised PDF) Action-pathway controls (Habitat 3.0; ranking on the 500 states × 32 candidates of Section 4.4).

(a) VariantF↑DRS↑Re-acq.↑Plan. ρ↑
R = [x<sub>ℓ+1</sub>, W<sub>a</sub>(a<sub>ℓ</sub>)], masked (paper)0.530.700.760.68
R = [x<sub>ℓ+1</sub>, W<sub>a</sub>(a<sub>ℓ</sub>)], no mask (paper)0.470.630.610.55
R = [x<sub>ℓ+1</sub>] only, masked (retrained)0.530.700.750.67
R = [x<sub>ℓ+1</sub>] only, no mask (retrained)0.470.630.620.56
Placebo mask: E blocked from the current-pose token, action edge kept (retrained)0.510.650.630.56
(b) Read-outFullW/o maskingGap
Evidence head on observation tokens (paper)0.680.550.13
GroundingDINO on the decoded view D(Ẑ)0.580.470.11
Separate latent head on Ẑ0.540.450.09
(c) ModelNearest 16Farthest 16All 32
Full0.640.650.68
W/o masking0.620.460.55
Adapted NWM0.510.380.47
AuthorsOpenReview ↗

Changes in the Revised Manuscript

New text is in blue.

  • Wording of the guarantee. Eq. 2 places the zero Jacobian on E<sub>ℓ+1</sub>. Sentences that read as stronger are reworded: Section 3.2.2 ("we give the action only the physical route", "Evidence thus follows the view", "The mask fixes the route"), the Figure 2 caption, the opening of Section 4, Section 4.4.3 ("it can only arise through the view") and Section 4.5 ("The action thus moves the view but not the target"). A sentence next to Eq. 3 says for which read-outs the factorization is exact.

  • Section 4.4: Table R2 (placebo mask, pose-only robot branch, read-outs from the latent, split by distance from the logged action).

  • Section 4.6: Table R1 (robot budget, running simulator, TrackVLA and a no-replanning control).

  • Table 1: numbers reported by the original papers and numbers from our reimplementation of TrackVLA++ carry separate marks. Table 2: TrackVLA++ row added, marked as our reimplementation.

  • Appendix A: the next-distance label and its statistics. Appendix C: how the TrackVLA++ baseline was obtained; timing of a planning cycle. Appendix D.2: visibility prediction split by starting state (Table R3), next to the pooled change-step number, which is kept; reward-term decomposition (Table R4); probes of E (Table R6). Appendix D.5: training on decorrelated logs (Table R7). Appendix D.6: action intervention at a fixed view (Table R8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.