- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
The paper proposes CST-WM, an action-conditioned world model for embodied visual tracking and target re-acquisition. Its state combines detector-derived target evidence, a visual latent, and the robot pose. The transition separates target-evidence, robot, and observation branches, using an attention mask to prevent the current action from directly entering the target-evidence branch. A detector-distilled evidence head scores imagined futures, and a CEM-based model-predictive controller selects actions.
Experiments on EVT-Bench and Habitat report improvements in following and recovery, supported by counterfactual action-ranking diagnostics and architectural ablations. Real-robot experiments provide additional evidence of applicability. My main concerns concern the positioning of the contribution, the justification of the representations and causal interpretation, and whether the recovery comparisons and computational settings sufficiently establish the method's practical advantage.
Critical strengths
S1. A meaningful problem with useful empirical evidence.
Maintaining visibility and recovering a moving target after occlusion are important problems for embodied tracking. Evaluating alternative actions through predicted future observations is a meaningful approach. The paper reports encouraging improvements across standard tracking, target-loss recovery, and cross-dataset transfer, together with a real-robot demonstration.
S2. The matched mask ablation provides evidence beyond overall task performance.
The comparison between the full model and the variant without masking is particularly informative. Removing the mask improves visibility prediction under logged actions, from 0.675 to 0.701 AUROC, but reduces counterfactual action-ranking correlation from 0.68 to 0.55 and closed-loop re-acquisition from 0.76 to 0.61. This distinction between predictive accuracy on logged actions and usefulness for planning is interesting. It supports the utility of the proposed constraint, although the precise explanation for that utility remains open.
Critical weaknesses
W1. The contribution and novelty require sharper positioning.
Predicting future observations or latent states, evaluating candidate actions, and selecting actions through model-predictive control are established ideas. Navigation World Models already uses action-conditioned visual prediction for planning, while WoMAP learns latent dynamics and detector-derived rewards for active perception. The latter is particularly relevant to the use of detection confidence, bounding-box size, and an evidence head for scoring imagined states.
The paper acknowledges these connections, but the distinction between established components, task-specific integration, and the proposed causal-routing contribution needs to be clearer. The principal methodological novelty appears to lie in the particular branch factorization and action mask. Its significance should therefore be established at that level, rather than inferred from the broader use of a world model for tracking and recovery.
W2. The representation design and its causal interpretation are insufficiently justified.
The state combines detector evidence H, a VAE latent Z, and robot pose x, but the paper does not sufficiently explain what information the learned target-evidence representation E captures, why the chosen compression and factorization are appropriate, or how they support the claimed separation of target dynamics, ego-motion, and observation changes. H provides confidence and apparent scale rather than target motion or identity, while Z is a distributed visual representation without an explicitly established physical interpretation. This matters particularly because the transition conditions on a single observation without a history or an explicit hidden-target belief.
The mask guarantees that E does not directly receive the current action. However, the robot branch retains an action embedding, and the evidence head reads observation-branch hidden tokens. Consequently, blocking the direct action-to-E connection does not by itself guarantee that action information influences predicted evidence exclusively through physically correct viewpoint changes. The reported gains support the architectural constraint, but do not fully distinguish the proposed causal explanation from a more general regularization or inductive-bias effect.
There is also a specific point requiring clarification: the action-independent E branch is supervised to predict whether the next relative robot–target distance is in range (Appendix A, Eq. 6), although that distance generally depends on the robot's current action. The compatibility of this supervision with the intended interpretation of E is unclear.
W3. Recovery comparisons and deployment-budget evidence do not yet fully establish the claimed advantage.
Table 2 compares recovery against TrackVLA and Adapted NWM. The matched NWM planner is a useful world-model baseline, but the table omits TrackVLA++, whose spatial reasoning and target memory directly address occlusion and target loss. OA-VAT also explicitly includes an occlusion-recovery planner. The paper explains that its Table 1 implementation uses the PID variant because the diffusion-planner checkpoint was unavailable. This is an understandable reproducibility limitation, but the comparison does not establish an advantage over OA-VAT's complete recovery approach.
The computational setting further limits interpretation. The default planner requires approximately 267 seconds per action on an RTX A6000, and simulation waits for planning to finish. The reported recovery times in simulation steps therefore exclude planning latency. The real robot uses a substantially smaller budget and achieves approximately 1.4 seconds per planning step on an offboard RTX 6000 Ada, which is valuable evidence and should be distinguished from the default cost.
However, the main mask ablations use the default budget. It remains unclear whether the same architectural advantage persists with the robot's shorter horizon and smaller candidate set, particularly when the target and robot continue moving during planning. A direct comparison under the deployment budget is important for connecting the central architectural claim to practical recovery.
Questions for authors
Q1. Contribution and novelty — corresponding to W1.
Please identify the principal methodological advance beyond NWM-style predictive planning and WoMAP-style detector-reward distillation. Which aspects are established components, which are adaptations to embodied tracking, and which constitute the new contribution? In particular, what new insight does the proposed factorization and action mask establish beyond a useful task-specific architectural constraint?
Q2. Representation and causal interpretation — corresponding to W2.
Please clarify what E is intended to represent, what supports that interpretation, and why the selected combination and compression of H, Z, and x is appropriate for tracking and recovery. How is the action-independent E branch's next-distance supervision in Eq. 6 consistent with that interpretation?
Please also distinguish the dependencies guaranteed by the mask from the physical or causal properties supported empirically. Since action information can still reach the evidence head through the robot and observation branches, what evidence distinguishes mitigation of the proposed action shortcut from a general regularization effect? A focused control on a representative subset, such as a comparably restrictive alternative mask or an evaluation that changes the logging-policy correlation, would help resolve this question.
Q3. Recovery comparisons and computational budget — corresponding to W3.
Please explain the selection of recovery baselines in Table 2, especially the omission of TrackVLA++ and the scope of conclusions possible without OA-VAT's complete recovery planner. If the stronger recovery baselines cannot be evaluated, please clarify the resulting limits of the comparative claims.
Can the full model and the existing variant without masking be compared under the robot deployment budget on a representative subset of the existing recovery episodes? Please report recovery performance and complete planning latency, and explain how the model's transition interval relates to the 1.4-second planning cycle, continued execution of the previous action, and observation staleness during planning.
6 comments
Response to Reviewer HEVQ Q2: Representation and Causal lInterpretation (W2)
What E represents, and why H, Z and x. Following your question on what E is intended to represent, we define as the target-related content of the current state, computed before the action is seen: whether the target is in view, where, and at what distance. To verify this reading, we conduct linear probes on the frozen E (Table R6). Target visibility is decoded with accuracy 0.92, in-range distance with 0.85, and the target's horizontal image position with R² 0.63. When the action is swapped at a fixed state, E does not follow it, while the unmasked branch does (R² 0.01 against 0.45). With the humanoid held still and the robot moving, E stays stable (0.98 against 0.61 for the adapted NWM, Table 3).
Table R6 (Table 18 in the revised PDF) Linear probes on the frozen (held-out Habitat 3.0 transitions).
| Probe target | Full | W/o masking |
|---|---|---|
| Target visible at ℓ+1 (accuracy) | 0.92 | 0.92 |
| Distance in following range at ℓ+1 (accuracy) | 0.85 | 0.88 |
| Target horizontal image position (R²) | 0.63 | 0.65 |
| Swapped action at a fixed state (R²) | 0.01 | 0.45 |
The state components each cover what the others cannot. H holds the two quantities the task is scored on, target in view and distance in range. Z holds the rest of the scene, the occluders and free space that determine what the camera sees after it moves. The pose x lets ego-motion follow kinematics instead of being learned. The scalar compression of H enters only as the transition's input. Replacing the trained signal with the raw detector score lowers DRS from 0.70 to 0.60 and re-acquisition from 0.76 to 0.66 (Table 10). Target identity and a hidden-target belief are outside our contribution, which concerns how the action reaches the target evidence; Appendix E states this boundary. Within it, recovery works by replanning at every step from a fresh detector reading. Only the last executed action carries over, through the smoothness limit of the admissible set.
Next-distance supervision in Eq. 6. Following your question on how this supervision is consistent with an action-free E, the answer is that learns the label averaged over the behavior policy's actions, . E can represent this average, and the mask's benefit does not rest on this loss. The label differs from the same label at step ℓ on 2.1% of training transitions. The loss cannot introduce an action dependence, because no gradient path from to E exists. Without , re-acquisition is 0.72; without the mask, 0.61 (Table 8). Details are in RrzZ Q1.
Guaranteed dependencies and evidence for shortcut mitigation. Following your question on which dependencies the mask guarantees and which are shown empirically, the split is as follows. Guaranteed: does not depend on (Eq. 2). Not guaranteed: that the head's reading depends on the action only through a physically correct view change. Action information reaches the observation tokens through the robot branch, as it must for the view to move. To separate shortcut mitigation from a general regularization effect, we conduct both controls you suggest and a third.
(1) Changing the correlation with the logging policy at training time (Table R7). We retrain the masked and the unmasked transition on Habitat 3.0 data re-collected with a fraction ε of the actions replaced by random admissible actions. As the correlation between logged action and target bearing falls from 0.55 to 0.05, the mask's advantage falls from 0.13 to 0.03 in Plan. ρ and from 0.15 to 0.02 in re-acquisition.
(2) A comparably restrictive alternative mask (Table R2a in the common response). A placebo mask removes the same number of attention edges from the target-evidence branch and keeps the action edge. It reaches re-acquisition 0.63 and Plan. ρ 0.56, slightly above no mask (0.61, 0.55) and below the action mask (0.76, 0.68).
(3) Changing the correlation at test time (Table R2c). The mask adds 0.02 to Plan. ρ on the 16 candidates nearest to the logged action sequence and 0.19 on the farthest 16.
Table R7 (Table 23 in the revised PDF) The mask's advantage as the logged action is decorrelated from the target. ε: fraction of logged actions replaced by random admissible ones.
| ε = 0 (paper) | ε = 0.5 | ε = 1.0 | |
|---|---|---|---|
| Correlation between action and target bearing | 0.55 | 0.31 | 0.05 |
| Plan. ρ↑, full | 0.68 | 0.66 | 0.62 |
| Plan. ρ↑, w/o masking | 0.55 | 0.60 | 0.59 |
| Plan. ρ↑, gap | 0.13 | 0.06 | 0.03 |
| Re-acq.↑, full | 0.76 | 0.73 | 0.65 |
| Re-acq.↑, w/o masking | 0.61 | 0.66 | 0.63 |
| Re-acq.↑, gap | 0.15 | 0.07 | 0.02 |
Response to Reviewer HEVQ Q3: Recovery Comparisons and Computational Budget (W3)
Recovery baselines in Table 2. Following your question on why Table 2 omits TrackVLA++ and what can be concluded without OA-VAT's complete recovery planner, we answer in two parts. The recovery claim holds against TrackVLA, TrackVLA++ and the adapted NWM. We make no claim over OA-VAT's planner. For the first part, we conduct the recovery experiment with our reimplementation of TrackVLA++ on the four target-loss conditions (Table R5).
TrackVLA++ has released neither code nor weights (arXiv 2510.07134; its project page links none), so its row in Table 1 has two sources, and the submission did not distinguish them.
- EVT-Bench columns: reported numbers. SR 86.0, TR 81.0 and CR 2.10 are the single-view results of Table I in the TrackVLA++ paper.
- Transfer columns on Habitat 3.0: our reimplementation. We built the model from the description in its paper. We started from the architecture and released code of TrackVLA and added the two TrackVLA++ modules, Polar-CoT spatial reasoning and the Target Identification Memory. Its training data follows the paper where we could obtain it; about 20% of that data could not be obtained or reproduced.
- Fidelity of the reimplementation. In our evaluation on EVT-Bench STT, it reaches SR 84.3, TR 80.2 and CR 2.26, against the reported 86.0, 81.0 and 2.10.
Table R5 (Table 2 in the revised PDF) Table 2 with TrackVLA++ added (Habitat 3.0 target-loss protocol). TTR is averaged over successfully re-acquired episodes only. Post-F and Rec-ES are in Table 2.
| Re-acq.↑ | Short occlusion | Long occlusion | Out-of-view drift | Distractor crossing |
|---|---|---|---|---|
| TrackVLA | 0.71 | 0.48 | 0.52 | 0.43 |
| TrackVLA++ (our reimpl.) | 0.73 | 0.57 | 0.57 | 0.51 |
| Adapted NWM | 0.75 | 0.54 | 0.57 | 0.49 |
| Ours | 0.84 | 0.69 | 0.73 | 0.65 |
| TTR↓ (steps) | Short occlusion | Long occlusion | Out-of-view drift | Distractor crossing |
|---|---|---|---|---|
| TrackVLA | 8.9 | 14.6 | 12.8 | 15.2 |
| TrackVLA++ (our reimpl.) | 8.39 | 12.73 | 11.83 | 13.87 |
| Adapted NWM | 8.1 | 13.2 | 11.7 | 14.0 |
| Ours | 6.4 | 9.8 | 8.7 | 10.6 |
TrackVLA++ improves on TrackVLA in every condition, most under long occlusion (0.57 against 0.48), where its target memory applies. Across short occlusion, long occlusion, out-of-view drift and distractor crossing, CST-WM stays ahead in re-acquisition by 0.11, 0.12, 0.16 and 0.14. Its mean TTR over each method's successful episodes is lower by 1.99, 2.93, 3.13 and 3.27 steps. The revised PDF marks reported and reimplemented numbers with separate symbols in Table 1 and describes the reimplementation in Appendix C.
On OA-VAT: the checkpoint of its diffusion planner is not released, so we ran its released pipeline with PID control (Appendix C). OA-VAT appears only in Table 1, and the revised PDF states this scope.
Full model against the unmasked variant at the deployment budget. The architectural advantage does not depend on the long horizon or the large candidate set, and it holds when the environment keeps moving during planning. Following your request, we conduct the comparison at the robot budget on 200 recovery episodes, 50 per condition (Table R1 in the common response). At the robot budget with the simulator paused, re-acquisition is 0.68 with the mask and 0.47 without. With the simulator running through each planning cycle, it is 0.55 and 0.42. TTR, which now counts the steps that pass during planning, is 11.67 and 15.11. The advantage is 0.21 with the simulator paused and 0.13 with it running, against 0.22 at the default budget.
Complete planning latency and timing. One planning step at the robot budget takes 1.4 s on the RTX 6000 Ada from the received observation to the issued action: 80 sequential denoising passes over 24 candidates, 8 evidence-head reads, one detector call and one encoder call (Table 20). One model transition spans one control step, Δt = 0.5 s, so the simulation rounds the cycle up to 3 control steps (1.5 s). The horizon T = 4 covers 2.0 s. During a cycle the robot repeats the action selected by the previous cycle. The plan is computed from the observation taken at the start of the cycle, so that observation is 1.5 s old when the first action is issued. This delay accounts for the full model's drop of 0.13 relative to the paused simulator (0.05 for the unmasked model).
Response to Reviewer HEVQ Q1: Contribution and Novelty (W1)
Thank you for the review.
We separate the contribution into three levels. The methodological advance is the action-routing constraint, together with the evidence that accuracy on logged futures can rank world models in the wrong order for planning.
Established components. An action-conditioned diffusion world model planned with CEM (NWM), and a reward head distilled from a detector that scores latent states without decoding (WoMAP). We use both and cite both where they enter (Sections 2 and 3.2.2).
Adaptation to tracking. The objective of Eq. 5 scores predicted visibility and apparent scale; it replaces NWM's goal image, which tracking does not have. The evidence head is trained jointly with the transition. Joint training reaches re-acquisition 0.76, against 0.43 for the same head on a frozen transition (Table 10).
New contribution. Our method introduces three new elements:
(1) A confound in logged tracking data. The behavior policy chooses its action according to where the target is, so a transition can predict next-step evidence from the action without modeling what the action does. In our logs the correlation between the action and the target bearing is 0.55, and the mask's advantage shrinks as this correlation is removed (Table R7,
HEVQ Q2).(2) A transition that removes the one edge this confound uses, aℓ→Eℓ+1, and keeps the action's effect through ego-motion. A placebo mask of the same size does not recover its gain (
HEVQ Q2).(3) A paired-intervention protocol that tests the difference where planning needs it, on actions the behavior policy did not take. It uses 500 saved states, each with 32 candidate sequences executed in the simulator and imagined by every model (Section 4.4). It locates the mask's gain on the candidates far from the logged action (
HEVQ Q2).Insight beyond a task-specific constraint. A world model that is more accurate under the logging policy is not necessarily better for planning. The unmasked model predicts logged futures better (visibility AUROC 0.701 against 0.675, Table 4), ranks counterfactual candidates worse (ρ 0.55 against 0.68) and recovers worse in closed loop (0.61 against 0.76). The adapted NWM, with 6.6× our parameters, does not rank better (ρ 0.47). We expect the same confound wherever a world model is trained on goal-directed logs, because an expert chooses its action based on the task-relevant state. The remedy we test is to compute the task-relevant representation before the action is read. The revised PDF states these three levels in the last paragraph of the introduction and in the related work.