- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes to use non-uniform multi-mask aggregation in a CWM. Specifically, it proposes two weighting methods: 1. APA, a training free weighting method using the NSG diffusion signal. 2. LPA, the work trains a tiny network on MOVi-F to predict the weights. In both cases, the weighted response is followed by windowed localization and one CWM re-evaluation. Besides, the paper introduces CMC, an evaluation protocol on dynamic tracks that counts missing predictions as failures. Experiments on subsets of DAVIS, Kinetics and RoboTAP show that APA and LPA have advances over the uniform weighting, while still behind CoTracker3 on most localization metrics.
Critical strengths
Treating aggregation as joint, permutation-invariant reliability estimation is a sensible formulation. The proposed method is lightweight and plug-in.
Critical weaknesses
- The baseline is a weakened (using 256 resolution without multiscale refinement) version of Opt-CWM, referred to as Uniform CWM in this paper. Opt-CWM achieves DAVIS-First AJ 47.53 [1] vs. 24.31 for Uniform CWM here. The evaluation subsets differ, but the headline relative gains appear to largely reflect a weak baseline. How much LPA and APA improve the complete Opt-CWM is still unclear. Besides, comparisons with other counterfactual methods are also missing, e.g., Point Prompting [2], PSI [3].
- LPA’s candidate-space EPE (2.41) is already close to the best-of-10 oracle (1.88), so the headroom of any reweighting scheme is bounded by the quality of the frozen candidates.
- CWMs are motivated by self-supervised motion estimation, whereas LPA is trained with Kubric groundtruth flow. When the Kubric groundtruth is available, why not just use a Kubric-trained tracker such as CoTracker3, which is clearly stronger in localization (DAVIS-First AJ 62.90 vs. 45.29)? The paper does not clarify what advantage LPA-CWM retains in this setting.
[1]Stojanov, Stefan, et al. "Self-supervised learning of motion concepts by optimizing counterfactuals." NeurIPS 2025.
[2]Shrivastava, Ayush, et al. "Point prompting: Counterfactual tracking with video diffusion models." ICLR 2026.
[3]Kotar, Klemen, et al. "World modeling with probabilistic structure integration." arXiv preprint arXiv:2509.09737, 2025.
Questions for authors
- Please report the gains of APA and LPA applied on the top of the complete Opt-CWM.
- Why is the new metric CMC necessary? Can you show concrete cases where CMC reveals failures that TAP-Vid AJ does not?
- Given that LPA relies on synthetic ground-truth motion, what advantage does LPA-CWM retain over Kubric-trained trackers?
4 comments
Response to Reviewer f6Rc (part 2)
We continue our responses to the remaining weaknesses and questions below.
- Question 3 / Weakness 3: What advantage remains when LPA uses synthetic supervision?
Answer: LPA improves motion extraction from a frozen counterfactual world model with a 3.0M-parameter reliability module, while offering complementary completeness and continuity advantages over CoTracker3. It learns to weight existing motion candidates rather than training a separate trajectory predictor; both the CWM predictor and intervention generator remain frozen. APA requires no trajectory supervision, and the existing LPA checkpoint transfers to complete Opt-CWM without retraining (Q1).
LPA is trained on MOVi-F and evaluated without dataset-specific fine-tuning. The main results from our original Table 1 are reproduced below:
| Dataset | Method | AJ ↑ | Cov. ↑ | [email protected] ↑ | DCA ↑ | CTS ↑ | VisBA ↑ | OF1 ↑ | LCC ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Kinetics | CoTracker3 | 50.86 | 87.06 | 79.67 | 50.15 | 46.61 | 78.00 | 58.56 | 80.41 |
| Kinetics | LPA-CWM | 41.91 | 94.65 | 89.97 | 51.92 | 58.25 | 82.97 | 62.67 | 86.84 |
| RoboTAP | CoTracker3 | 56.58 | 81.02 | 73.92 | 51.40 | 43.05 | 73.11 | 59.35 | 75.21 |
| RoboTAP | LPA-CWM | 47.42 | 92.00 | 86.88 | 50.26 | 54.22 | 78.03 | 61.00 | 85.49 |
CoTracker3 remains stronger in AJ, while LPA-CWM provides better completeness and continuity on both datasets. These are complementary benefits, rather than evidence that LPA-CWM replaces a general-purpose tracker. The scaled CoTracker3 checkpoint used here additionally trains on real-video pseudo-labels, so the supervision budgets are not matched.
- Question 4 / Weakness 2: LPA’s candidate-space EPE (2.41) is already close to the best-of-10 oracle (1.88), so the headroom of any reweighting scheme is bounded by the quality of the frozen candidates.
Answer: Reweighting is limited by frozen candidate quality, but LPA substantially improves how the same candidates are used. As acknowledged in Section 5, missing or systematically incorrect motion evidence requires better candidates. Fig. 6 compares aggregation on the same ten candidates from MOVi-F validation videos:
| Aggregation | Candidate-space EPE (px) ↓ |
|---|---|
| Uniform weighting | 5.48 |
| APA | 3.80 |
| LPA | 2.41 |
| Ground-truth best-of-10 reference | 1.88 |
LPA reduces EPE substantially without changing the candidate budget or adapting the CWM, approaching the best-of-10 reference that requires ground truth unavailable at inference. The new complete Opt-CWM results in Question 1 further show that better candidate generation and learned aggregation are complementary.
We have incorporated your suggestions, together with those of the other reviewers, into the revised 10-page main text and appendix. Substantive revisions are highlighted in blue in the revised PDF.
-
Question 1 / Weakness 1: Opt-CWM and counterfactual comparisons. Section 4.1 (p. 6) distinguishes the controlled baseline from high-resolution multiscale Opt-CWM. Section 4.6 and Table 4 (pp. 9–10) report gains without retraining, including AJ 45.61 to 47.01 with LPA; Appendix E.4, Table E.3 (p. 25), gives full results. Table 1 (p. 8) includes PSI-0.5, while Section 2 (p. 3) discusses Point Prompting’s different interface.
-
Question 2: CMC beyond AJ. Section 4.3 and Figure 6 (pp. 7–8), with Appendix C and Table C.3 (pp. 20–21), show identical AJ and coverage but different uninterrupted prediction runs, providing a concrete continuity diagnostic.
-
Question 3 / Weakness 3: Advantages retained with synthetic supervision. Section 1 (p. 2) clarifies that only the 3.0M-parameter reliability module is trained, while the CWM predictor and intervention generator remain frozen. Without dataset-specific fine-tuning, LPA-CWM achieves higher coverage, [email protected], and LCC than CoTracker3 on Kinetics and RoboTAP, despite lower AJ (Table 1, p. 8). The same checkpoint also transfers to Opt-CWM without retraining (Section 4.6, Table 4, pp. 9–10).
-
Question 4 / Weakness 2: Dependence on candidate quality. Figure 7 (p. 9) and Appendix E.1 (pp. 22–23) analyze aggregation on fixed candidates; Section 5 (p. 10) explicitly acknowledges the limitation imposed by missing or systematically incorrect candidates.
Thank you for helping us strengthen the comparisons and clarify the contribution. Should our responses and the revised paper resolve your concerns, we would be grateful if you could consider revising your score upward.
Thank you for the response and additional experiments, though my concerns largely remain.
- LPA/APA improves Opt-CWM by 1.4/0.9 AJ, which is minor and just comparable with simply tuning the mask ratio of Opt-CWM (shown in its Table 6).
- I see the point of CMC is temporal completeness/interruptions. However, there are existing metrics consider these failures, e.g., Survival [1]. The necessity of introducing a new metric is still unclear. Furthermore, point trackers like CoTracker3 can increase the temporal completeness by adjusting the visibility/confidence threshold.
- The advantage is not worth the cost: introducing supervised learning to a self-supervised method, but provides only minor gains, and still behind the fully supervised methods.
Therefore I'll keep my score.
[1] Zheng, Yang, et al. "Pointodyssey: A large-scale synthetic dataset for long-term point tracking." ICCV, 2023.
Response to Reviewer f6Rc (part 1)
Thank you for the constructive feedback. We address the concerns below.
Answer: APA and LPA improve the complete Opt-CWM without additional training or fine-tuning, and LPA also outperforms the released PSI-0.5 model on our DAVIS subset. We evaluate complete Opt-CWM (512 × 512, MM=10, MS=4), changing only candidate weighting and reusing the existing LPA checkpoint. Our original 256-resolution baseline omits multiscale refinement, so its AJ is not directly comparable to the published Opt-CWM result.
TAP-Vid results
Distances are in pixels; other values are percentages. Bold indicates the best result.
CMC results
All values are percentages; CTS denotes CTS@(4,0.8).
LPA increases complete Opt-CWM's AJ from 45.61 to 47.01 and improves all seven CMC measures. Thus, the gains persist when the existing module is transferred directly to the stronger baseline.
PSI-0.5 uses the released model on the same subset, with forward–backward visibility checking at 6 native-frame pixels, without test-label tuning. Point Prompting propagates inserted markers through generated video; our method aggregates existing CWM responses. Its project page currently provides no implementation link, so a matched quantitative comparison remains unavailable.
Answer: CMC reveals trajectory completeness and temporal interruptions that an AJ score alone cannot distinguish. As Reviewer GHnp also notes in Critical Strength 2, “The proposed CMC protocol provides a useful complementary perspective on motion correspondence by jointly assessing localization, trajectory completeness, visibility, and temporal continuity.”
AJ penalizes missing predictions but does not distinguish their temporal arrangement. In a real DAVIS gold-fish track, both methods return four of six visible sampled observations and have identical AJ counts at all five thresholds:
The same AJ and coverage conceal different temporal structures: Opt-CWM is interrupted, whereas LPA forms a longer uninterrupted run. This is continuity over the sampled observations, not complete recovery of the track.
Our original Table 1 also shows complementary performance dimensions at the dataset level:
On Kinetics, LPA-CWM has lower AJ but higher scores on all seven CMC measures. On DAVIS, SMURF has extensive coverage and continuity but poor localization, exposed by its low DCA and CTS. CMC's dimensions should therefore be read jointly: they separate incomplete output, inaccurate correspondence, and temporal interruption rather than replacing AJ.