acceptodds
Under review as a conference paper at ICLR 2027

LPA-CWM: A Learned Physical Adjudicator for Motion Reasoning with Counterfactual World Models

Abstract

Counterfactual world models (CWM) extract motion from pretrained video predictors by comparing factual and intervened predictions, but uniform aggregation weights responses equally without explicitly incorporating physical priors. Our key insight is to incorporate physical priors into candidate reliability learning, motivating **LPA-CWM** with a lightweight Learned Physical Adjudicator (LPA). Trained on dense MOVi-F trajectories, the 3.0M-parameter LPA compares visual context and response structure across an unordered candidate set to predict relative weights; windowed localization and one paired re-evaluation recover motion with the CWM frozen. Existing video-level benchmarks do not directly assess motion correspondence, where low localization error can conceal missing trajectory segments. We introduce Completeness-aware Motion Correspondence (CMC), a ground-truth-anchored protocol jointly measuring localization, completeness, visibility, and continuity, counting missing predictions as failures on visible dynamic points. Across DAVIS, Kinetics, and RoboTAP, LPA-CWM improves all main CMC measures over Uniform CWM, with relative gains of 18.1%–60.0% in average Dynamic Correspondence Accuracy (), and improves TAP-Vid First tracking accuracy (overview: https://LPA-CWM.github.io).

Primary area:
applications to robotics, autonomy, planning
Keywords:
counterfactual world models, motion correspondence, candidate reliability, point tracking, motion reasoning, world models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f6RcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to use non-uniform multi-mask aggregation in a CWM. Specifically, it proposes two weighting methods: 1. APA, a training free weighting method using the NSG diffusion signal. 2. LPA, the work trains a tiny network on MOVi-F to predict the weights. In both cases, the weighted response is followed by windowed localization and one CWM re-evaluation. Besides, the paper introduces CMC, an evaluation protocol on dynamic tracks that counts missing predictions as failures. Experiments on subsets of DAVIS, Kinetics and RoboTAP show that APA and LPA have advances over the uniform weighting, while still behind CoTracker3 on most localization metrics.

Critical strengths

Treating aggregation as joint, permutation-invariant reliability estimation is a sensible formulation. The proposed method is lightweight and plug-in.

Critical weaknesses
  • The baseline is a weakened (using 256 resolution without multiscale refinement) version of Opt-CWM, referred to as Uniform CWM in this paper. Opt-CWM achieves DAVIS-First AJ 47.53 [1] vs. 24.31 for Uniform CWM here. The evaluation subsets differ, but the headline relative gains appear to largely reflect a weak baseline. How much LPA and APA improve the complete Opt-CWM is still unclear. Besides, comparisons with other counterfactual methods are also missing, e.g., Point Prompting [2], PSI [3].
  • LPA’s candidate-space EPE (2.41) is already close to the best-of-10 oracle (1.88), so the headroom of any reweighting scheme is bounded by the quality of the frozen candidates.
  • CWMs are motivated by self-supervised motion estimation, whereas LPA is trained with Kubric groundtruth flow. When the Kubric groundtruth is available, why not just use a Kubric-trained tracker such as CoTracker3, which is clearly stronger in localization (DAVIS-First AJ 62.90 vs. 45.29)? The paper does not clarify what advantage LPA-CWM retains in this setting.

[1]Stojanov, Stefan, et al. "Self-supervised learning of motion concepts by optimizing counterfactuals." NeurIPS 2025.

[2]Shrivastava, Ayush, et al. "Point prompting: Counterfactual tracking with video diffusion models." ICLR 2026.

[3]Kotar, Klemen, et al. "World modeling with probabilistic structure integration." arXiv preprint arXiv:2509.09737, 2025.

Questions for authors
  • Please report the gains of APA and LPA applied on the top of the complete Opt-CWM.
  • Why is the new metric CMC necessary? Can you show concrete cases where CMC reveals failures that TAP-Vid AJ does not?
  • Given that LPA relies on synthetic ground-truth motion, what advantage does LPA-CWM retain over Kubric-trained trackers?
4 comments
AuthorsOpenReview ↗

Response to Reviewer f6Rc (part 1)

Thank you for the constructive feedback. We address the concerns below.

  • Question 1 / Weakness 1: Gains on complete Opt-CWM and additional counterfactual baselines

Answer: APA and LPA improve the complete Opt-CWM without additional training or fine-tuning, and LPA also outperforms the released PSI-0.5 model on our DAVIS subset. We evaluate complete Opt-CWM (512 × 512, MM=10, MS=4), changing only candidate weighting and reusing the existing LPA checkpoint. Our original 256-resolution baseline omits multiscale refinement, so its AJ is not directly comparable to the published Opt-CWM result.

TAP-Vid results

Distances are in pixels; other values are percentages. Bold indicates the best result.

MethodAJ ↑⟨δ_avg⟩ ↑Avg. distance ↓Median distance ↓OA ↑
PSI-0.531.4851.4918.576.9654.09
Opt-CWM45.6162.1010.323.0680.21
Opt-CWM + APA46.5462.829.882.8881.32
Opt-CWM + LPA47.0163.548.572.7881.69

CMC results

All values are percentages; CTS denotes CTS@(4,0.8).

MethodCov. ↑[email protected] ↑DCA ↑CTS ↑VisBA ↑OF1 ↑LCC ↑
PSI-0.543.1517.1632.1614.5166.0442.2939.26
Opt-CWM83.7572.8056.8349.7771.9849.5677.91
Opt-CWM + APA84.5675.7157.5352.0074.8552.5178.92
Opt-CWM + LPA85.2976.0958.1752.3474.8952.6779.99

LPA increases complete Opt-CWM's AJ from 45.61 to 47.01 and improves all seven CMC measures. Thus, the gains persist when the existing module is transferred directly to the stronger baseline.

PSI-0.5 uses the released model on the same subset, with forward–backward visibility checking at 6 native-frame pixels, without test-label tuning. Point Prompting propagates inserted markers through generated video; our method aggregates existing CWM responses. Its project page currently provides no implementation link, so a matched quantitative comparison remains unavailable.

  • Question 2: Why is the new metric CMC necessary? Can you show concrete cases where CMC reveals failures that TAP-Vid AJ does not?

Answer: CMC reveals trajectory completeness and temporal interruptions that an AJ score alone cannot distinguish. As Reviewer GHnp also notes in Critical Strength 2, “The proposed CMC protocol provides a useful complementary perspective on motion correspondence by jointly assessing localization, trajectory completeness, visibility, and temporal continuity.”

AJ penalizes missing predictions but does not distinguish their temporal arrangement. In a real DAVIS gold-fish track, both methods return four of six visible sampled observations and have identical AJ counts at all five thresholds:

MethodTrack-level AJ ↑Coverage ↑Consecutive returned-run lengthsLCC ↑
Opt-CWM42.2266.672 + 233.33
Opt-CWM + LPA42.2266.67466.67

The same AJ and coverage conceal different temporal structures: Opt-CWM is interrupted, whereas LPA forms a longer uninterrupted run. This is continuity over the sampled observations, not complete recovery of the track.

Our original Table 1 also shows complementary performance dimensions at the dataset level:

DatasetMethodAJ ↑Cov. ↑[email protected] ↑DCA ↑CTS ↑VisBA ↑OF1 ↑LCC ↑
KineticsCoTracker350.8687.0679.6750.1546.6178.0058.5680.41
KineticsLPA-CWM41.9194.6589.9751.9258.2582.9762.6786.84
DAVISSMURF7.3991.9885.5411.252.6756.6414.9283.29
DAVISSEA-RAFT40.7257.7631.2543.0127.4365.7839.5654.07

On Kinetics, LPA-CWM has lower AJ but higher scores on all seven CMC measures. On DAVIS, SMURF has extensive coverage and continuity but poor localization, exposed by its low DCA and CTS. CMC's dimensions should therefore be read jointly: they separate incomplete output, inaccurate correspondence, and temporal interruption rather than replacing AJ.

AuthorsOpenReview ↗

Response to Reviewer f6Rc (part 2)

We continue our responses to the remaining weaknesses and questions below.

  • Question 3 / Weakness 3: What advantage remains when LPA uses synthetic supervision?

Answer: LPA improves motion extraction from a frozen counterfactual world model with a 3.0M-parameter reliability module, while offering complementary completeness and continuity advantages over CoTracker3. It learns to weight existing motion candidates rather than training a separate trajectory predictor; both the CWM predictor and intervention generator remain frozen. APA requires no trajectory supervision, and the existing LPA checkpoint transfers to complete Opt-CWM without retraining (Q1).

LPA is trained on MOVi-F and evaluated without dataset-specific fine-tuning. The main results from our original Table 1 are reproduced below:

DatasetMethodAJ ↑Cov. ↑[email protected] ↑DCA ↑CTS ↑VisBA ↑OF1 ↑LCC ↑
KineticsCoTracker350.8687.0679.6750.1546.6178.0058.5680.41
KineticsLPA-CWM41.9194.6589.9751.9258.2582.9762.6786.84
RoboTAPCoTracker356.5881.0273.9251.4043.0573.1159.3575.21
RoboTAPLPA-CWM47.4292.0086.8850.2654.2278.0361.0085.49

CoTracker3 remains stronger in AJ, while LPA-CWM provides better completeness and continuity on both datasets. These are complementary benefits, rather than evidence that LPA-CWM replaces a general-purpose tracker. The scaled CoTracker3 checkpoint used here additionally trains on real-video pseudo-labels, so the supervision budgets are not matched.

  • Question 4 / Weakness 2: LPA’s candidate-space EPE (2.41) is already close to the best-of-10 oracle (1.88), so the headroom of any reweighting scheme is bounded by the quality of the frozen candidates.

Answer: Reweighting is limited by frozen candidate quality, but LPA substantially improves how the same candidates are used. As acknowledged in Section 5, missing or systematically incorrect motion evidence requires better candidates. Fig. 6 compares aggregation on the same ten candidates from MOVi-F validation videos:

AggregationCandidate-space EPE (px) ↓
Uniform weighting5.48
APA3.80
LPA2.41
Ground-truth best-of-10 reference1.88

LPA reduces EPE substantially without changing the candidate budget or adapting the CWM, approaching the best-of-10 reference that requires ground truth unavailable at inference. The new complete Opt-CWM results in Question 1 further show that better candidate generation and learned aggregation are complementary.

We have incorporated your suggestions, together with those of the other reviewers, into the revised 10-page main text and appendix. Substantive revisions are highlighted in blue in the revised PDF.

  • Question 1 / Weakness 1: Opt-CWM and counterfactual comparisons. Section 4.1 (p. 6) distinguishes the controlled baseline from high-resolution multiscale Opt-CWM. Section 4.6 and Table 4 (pp. 9–10) report gains without retraining, including AJ 45.61 to 47.01 with LPA; Appendix E.4, Table E.3 (p. 25), gives full results. Table 1 (p. 8) includes PSI-0.5, while Section 2 (p. 3) discusses Point Prompting’s different interface.

  • Question 2: CMC beyond AJ. Section 4.3 and Figure 6 (pp. 7–8), with Appendix C and Table C.3 (pp. 20–21), show identical AJ and coverage but different uninterrupted prediction runs, providing a concrete continuity diagnostic.

  • Question 3 / Weakness 3: Advantages retained with synthetic supervision. Section 1 (p. 2) clarifies that only the 3.0M-parameter reliability module is trained, while the CWM predictor and intervention generator remain frozen. Without dataset-specific fine-tuning, LPA-CWM achieves higher coverage, [email protected], and LCC than CoTracker3 on Kinetics and RoboTAP, despite lower AJ (Table 1, p. 8). The same checkpoint also transfers to Opt-CWM without retraining (Section 4.6, Table 4, pp. 9–10).

  • Question 4 / Weakness 2: Dependence on candidate quality. Figure 7 (p. 9) and Appendix E.1 (pp. 22–23) analyze aggregation on fixed candidates; Section 5 (p. 10) explicitly acknowledges the limitation imposed by missing or systematically incorrect candidates.

Thank you for helping us strengthen the comparisons and clarify the contribution. Should our responses and the revised paper resolve your concerns, we would be grateful if you could consider revising your score upward.

Thank you for the response and additional experiments, though my concerns largely remain.

  1. LPA/APA improves Opt-CWM by 1.4/0.9 AJ, which is minor and just comparable with simply tuning the mask ratio of Opt-CWM (shown in its Table 6).
  2. I see the point of CMC is temporal completeness/interruptions. However, there are existing metrics consider these failures, e.g., Survival [1]. The necessity of introducing a new metric is still unclear. Furthermore, point trackers like CoTracker3 can increase the temporal completeness by adjusting the visibility/confidence threshold.
  3. The advantage is not worth the cost: introducing supervised learning to a self-supervised method, but provides only minor gains, and still behind the fully supervised methods.

Therefore I'll keep my score.

[1] Zheng, Yang, et al. "Pointodyssey: A large-scale synthetic dataset for long-term point tracking." ICCV, 2023.

Reviewer_kdjQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a method to improve motion tracking using pretrained Counterfactual World Models (CWMs), by learning a weighting mechanism for the candidate trajectories generated by CWM. Existing CWM models can produce inconsistent motion estimates and uniformly averaging them may reduce tracking accuracy. The paper addresses this limitation by replacing the uniform weighting with learning the weights for multiple motion candidates by training a Learned Physical Adjudicator (LPA) which is a lightweight 3M-parameter neural network. This network is trained on synthetic MOVi-F trajectories by finding a loss between prediction and ground-truth endpoint errors. The underlying CWM is kept frozen. The paper also discusses about APA-CWM which is a training-free alternative that assigns weights by an analytic spatiotemporal consistency measure. The paper also introduces a Completeness-aware Motion Correspondence (CMC) metric that can be used to measure tracking localization, completeness, visibility, and temporal continuity. Evaluations are conducted on DAVIS, Kinetics, and RoboTAP and the proposed method, LPA-CWM performs better than Uniform CWM and APA-CWM across the proposed CMC metric and TAP-VID first metric.

Overall, the contributions of the paper include 1) learning a weighting mechanism for the candidate trajectories to improve motion tracking, 2) an evaluation metric to compute the tracking localization, completeness, visibility, and temporal continuity.

Critical strengths
  1. The method has been evaluated on diverse datasets, including everyday human activities in Kinetics, general object motion in DAVIS, and robotic tasks in RoboTAP. LPA is trained on synthetic MOVi-F trajectories and it consistently shows improvement in motion tracking on real-world datasets without requiring dataset-specific adaptation. This shows good generalization across different types of videos and motion scenarios.

  2. LPA has only 3 million trainable parameters, while the underlying pretrained CWM remains frozen. This has good practical advantage because the method is able to improve motion tracking without requiring expensive retraining or fine-tuning of the world model.

Critical weaknesses
  1. The paper claims that LPA learns physical motion priors (Line 074). However, the loss function is designed around the pixel distance between predicted and ground-truth endpoints, where candidates with smaller endpoint errors/distances receive higher target weights. I'm not sure if minimizing pixel-level tracking errors can be correlated with learning physical priors. Learning physical priors may involve understanding gravity, momentum, object interactions, or estimating material properties, but pixel distance learns about visual+motion information that improves tracking accuracy but it does not learn physics. The paper shows improvements in tracking accuracy but I'm not sure if these results can establish that the model has learned physical priors. It has rather learned better visual or motion-based information.

  2. The paper claims that different target-frame masks can produce inconsistent CWM predictions and that uniformly averaging them can lead to poor tracking. Table 1 shows that LPA-CWM has better performance than Uniform CWM. However, the paper does not thoroughly quantify/investigate that how frequently inconsistent candidate predictions occur and whether the degree of disagreement is correlated with tracking failures and it is the disagreement among the candidates which causes poor tracking.

  3. From the paper, it is still less convincing as to why a new evaluation protocol is necessary when existing TAP-Vid metrics already combine localization and visibility. Line 087 mentions that CMC evaluates completeness and temporal continuity. However, these properties do not necessarily indicate accurate motion. For example, a model may continuously predict a point as visible, even in the wrong locations, and still achieve high completeness and continuity scores.

Questions for authors

I am not convinced that motion reasoning is the most appropriate term to use. The approach primarily learns weights for candidate motion predictions rather than explicitly reasoning about object motion, physical interactions, or causal relationships between events. Instead of motion reasoning, simply using motion tracking can be useful as the paper does show results and improvement in the motion tracking accuracy.

7 comments
AuthorsOpenReview ↗

Response to Reviewer kdjQ (part 1)

Thank you for the constructive feedback. We address the questions and weaknesses below.

  • Question 1: Is "motion reasoning" an appropriate term for this work?

Answer: We agree that our direct empirical contribution is improved motion tracking; specifically, LPA improves the interpretation and aggregation of counterfactual motion responses from a frozen CWM.

(1) Tracking gains are supported by both TAP-Vid and CMC localization metrics. We fully agree with the reviewer that LPA learns candidate reliability weights rather than explicitly modeling physical interactions or event-level causal relationships. In our new DAVIS experiments, we apply APA and the existing LPA checkpoint to complete Opt-CWM without additional training or fine-tuning, using identical evaluation settings:

MethodTAP-Vid AJ ↑CMC DCA ↑CMC CTS ↑
Opt-CWM45.6156.8349.77
Opt-CWM + APA46.5457.5352.00
Opt-CWM + LPA47.0158.1752.34

The improvements in AJ and DCA demonstrate better localization, while CTS additionally requires sufficient trajectory coverage. All values are percentages; CTS denotes CTS@(4,0.8).

(2) LPA improves the use of counterfactual evidence within the CWM's motion-inference process. The underlying CWM predicts how the target frame changes when a local perturbation is applied at the source query. Comparing predictions with and without this perturbation reveals the target locations that respond to the queried region. Different target-frame masks produce multiple such counterfactual responses; LPA judges their relative reliability and aggregates them. Thus, the weights operate on motion evidence obtained through interventions in the frozen world model.

Our original Fig. 6 isolates aggregation on the same ten candidates from MOVi-F validation videos:

AggregationCandidate-space EPE (px) ↓
Uniform weighting5.48
APA3.80
LPA2.41
Ground-truth best-of-10 reference1.88

These results show that LPA extracts more accurate motion estimates from the available counterfactual responses. Here, motion reasoning refers specifically to assessing and combining intervention-derived motion evidence, without claiming that LPA explicitly models physical interactions or event-level causal relationships.

  • Question 2: Does endpoint-error supervision learn physical knowledge or correspondence confidence?

Answer: We agree that endpoint-error supervision teaches LPA more effective visual and motion reliability information, rather than directly learning gravity, momentum, or material properties.

(1) LPA learns the relative reliability of motion candidates. As the reviewer observes, our supervision assigns higher target weights to candidates with smaller distances to the ground-truth endpoint. This guides LPA to use visual and motion information to identify reliable candidates, rather than explicitly estimating physical quantities such as gravity, momentum, or material properties.

(2) Our method builds on APA, an analytic module incorporating a physics-inspired prior. The prior used by APA draws on NSG, which relates spatial structure to temporal changes through a probability-mass continuity equation, analogous to mass conservation in fluid mechanics. APA uses this spatiotemporal consistency prior to assign analytic candidate weights, providing the starting point for our learnable aggregation approach.

Our new DAVIS experiments apply APA and the existing LPA checkpoint to complete Opt-CWM without additional training or fine-tuning, under the same evaluation settings:

TAP-Vid results

MethodAJ ↑⟨δ_avg⟩ ↑AD ↓MD ↓OA ↑
Opt-CWM45.6162.1010.323.0680.21
Opt-CWM + APA46.5462.829.882.8881.32
Opt-CWM + LPA47.0163.548.572.7881.69

CMC results

MethodCov. ↑[email protected] ↑DCA ↑CTS ↑VisBA ↑OF1 ↑LCC ↑
Opt-CWM83.7572.8056.8349.7771.9849.5677.91
Opt-CWM + APA84.5675.7157.5352.0074.8552.5178.92
Opt-CWM + LPA85.2976.0958.1752.3474.8952.6779.99

AD and MD are in pixels; other values are percentages. CTS denotes CTS@(4,0.8).

The analytic APA module incorporating this physics-inspired prior already improves performance across the reported measures. Building on this aggregation framework, LPA provides a learnable reliability estimator that further improves performance.

AuthorsOpenReview ↗

Response to Reviewer kdjQ (part 2)

We continue our responses to the remaining weaknesses below.

  • Question 3: How frequently inconsistent candidate predictions occur and whether the degree of disagreement is correlated with tracking failures and it is the disagreement among the candidates which causes poor tracking?

Answer: Large candidate disagreements occur frequently in our diagnostic sample, and uniform aggregation has a higher localization failure rate when candidate disagreement is large, resulting in poorer tracking; when accurate candidates are available, LPA can correct some of these aggregation failures, thereby improving tracking on those queries.

(1) Quantifying the frequency of candidate disagreement. We measure disagreement as the RMS distance of the ten candidate peaks from their centroid, on the 256-pixel evaluation grid. We use ≤4px for small disagreement and >16px for large disagreement: 16px is four times our localization tolerance, separating substantial dispersion from small positional differences. A prediction with endpoint error ≥4px is counted as a localization failure.

We evaluate a stratified sample of 500 visible frame pairs from DAVIS.

Visible frame pairsLarge-disagreement proportionUniform failure rate: small disagreementUniform failure rate: large disagreement
50035.80% (179/500)14.05% (26/185)79.33% (142/179)

(2) Larger disagreements are associated with more tracking failures. Uniform's final localization failure rate rises from 14.05% to 79.33% between the small- and large-disagreement groups. These results show a clear association between candidate disagreement and tracking failure: greater disagreement is accompanied by a higher failure rate.

(3) Uniform aggregation can fail despite having accurate candidates. Among the same 179 large-disagreement pairs, 68 contain at least one candidate peak within 4px of the ground-truth endpoint. Uniform's initial aggregation nevertheless fails on 47 of these pairs. Keeping the same initial candidates, LPA corrects 11 of those failures:

Large-disagreement pairs containing an accurate candidateUniform aggregation failuresFailures corrected by LPA
37.99% (68/179)69.12% (47/68)23.40% (11/47)

These results show that LPA can recover accurate motion evidence missed by uniform aggregation and correct its localization failures, improving tracking on the recovered queries.

  • Question 4: Why is CMC needed beyond TAP-Vid, and how does it distinguish accurate from continuous tracking?

Answer: CMC jointly evaluates localization, trajectory completeness, visibility, and temporal continuity to distinguish inaccurate correspondence from missing or interrupted tracking, providing a more comprehensive diagnosis of motion-estimation accuracy.

As Reviewer GHnp also notes in Critical Strength 2, “The proposed CMC protocol provides a useful complementary perspective on motion correspondence by jointly assessing localization, trajectory completeness, visibility, and temporal continuity.”

(1) We agree that completeness and continuity alone do not guarantee accurate localization. We observe exactly this behavior in the original Table 1 results on DAVIS:

MethodAJ ↑Coverage ↑[email protected] ↑DCA ↑CTS ↑LCC ↑
SMURF7.3991.9885.5411.252.6783.29
SEA-RAFT40.7257.7631.2543.0127.4354.07

SMURF returns predictions over more of the visible trajectory and has higher completeness and continuity, but substantially poorer localization. AJ correctly reflects its lower accuracy; CMC further identifies the failure pattern as extensive coverage with inaccurate correspondence, through the low DCA and CTS scores.

(2) CMC's four dimensions should be interpreted jointly to identify different tracking failures. Coverage, [email protected], and LCC describe coverage and temporal structure; DCA measures localization accuracy, while CTS requires both sufficient coverage and accurate correspondence. Thus, continuously predicting incorrect locations does not yield consistently strong CMC results.

CMC also distinguishes temporal structures that can have identical AJ. In a real DAVIS gold-fish track, both methods return four of six visible sampled observations and have identical AJ counts at all five thresholds:

MethodTrack-level AJ ↑Coverage ↑LCC ↑
Opt-CWM42.2266.6733.33
Opt-CWM + LPA42.2266.6766.67

Opt-CWM's returned predictions form two runs of length two, whereas LPA forms one uninterrupted run of length four. The same AJ and coverage therefore conceal different temporal continuity. By distinguishing localization errors, incomplete coverage, and interruptions, CMC offers more specific guidance on which aspects of motion tracking need improvement.

Following your comments and the feedback from the other reviewers, we have updated the 10-page main text and the supporting appendix. Substantive changes are shown in blue in the revised PDF.

  • Question 1: Task terminology. Section 5 (p. 10) now describes the task as “motion correspondence recovery,” consistent with the focus on reliable motion readout from a frozen CWM clarified in Section 1 (p. 2).

  • Question 2 / Weakness 1: Physical motivation and learned reliability. Section 1 (p. 2) distinguishes APA’s physics-inspired motivation from LPA’s supervised learning of candidate reliability.

  • Question 3 / Weakness 2: Disagreement and aggregation failures. Section 4.4 and Table 2 (p. 9), with details in Appendix E.3 (p. 24), quantify disagreement on 500 visible query–target pairs, its sample-level association with failures, and recoverable aggregation failures using the same candidates.

  • Question 4 / Weakness 3: What CMC adds beyond AJ. Section 4.2 (p. 7) explains why coverage and continuity must be interpreted jointly with localization. Section 4.3 and Figure 6 (pp. 7–8), supported by Appendix C and Table C.3 (pp. 20–21), show identical AJ (42.22%) but different temporal continuity on the same DAVIS track.

We appreciate your feedback, which helped us make the scope and evaluation more precise. If our replies and revisions address your concerns, we hope you will consider revising your score upward.

Reviewer_GHnpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LPA-CWM to improve motion correspondence from a frozen counterfactual world model by learning reliability weights for mask-conditioned responses. A lightweight 3.0M-parameter adjudicator is trained on MOVi-F trajectories, followed by weighted aggregation, windowed localization, and CWM re-evaluation. The authors also introduce an analytic weighting baseline (APA-CWM) and a completeness-aware evaluation protocol. Experiments on DAVIS, Kinetics, and RoboTAP show consistent improvements over uniform and analytic aggregation under both CMC and TAP-Vid First metrics, supported by ablation studies.

Critical strengths
  1. The paper introduces a learned candidate-reliability weighting framework that improves motion correspondence without modifying the pretrained CWM.
  2. The proposed CMC protocol provides a useful complementary perspective on motion correspondence by jointly assessing localization, trajectory completeness, visibility, and temporal continuity.
  3. LPA-CWM achieves consistent improvements across three datasets on both CMC and TAP-Vid First metrics. Well-controlled comparisons and ablation studies further support the effectiveness of the proposed components.
Critical weaknesses
  1. LPA is supervised by endpoint errors and relies primarily on visual and response features, without explicitly modeling physical constraints. It remains unclear whether the learned weights capture physical knowledge or merely correspondence confidence.
  2. Although the ablations validate individual components, the paper does not compare against simpler learned reliability estimators, such as an MLP-based candidate scorer. This makes it difficult to assess whether the proposed architecture is necessary for the reported gains.
  3. It remains unclear whether the learned weights capture generalizable counterfactual evidence or protocol-specific response patterns.
Questions for authors
  1. Since LPA is trained on adjacent MOVi-F frames, how does its performance vary with increasing temporal gaps, large motion, or prolonged occlusion during evaluation?
  2. How much of the final improvement comes from learned candidate weighting versus the subsequent CWM re-evaluation? A matched comparison across different re-evaluation budgets would help clarify their respective contributions.
  3. What is the additional inference overhead of LPA compared with Uniform CWM and APA-CWM under identical settings? How does the method scale with the number of candidates and tracked points?
8 comments
AuthorsOpenReview ↗

Response to Reviewer GHnp (part 1)

Thank you for the constructive feedback. We address the questions and weaknesses below.

  • Question 1: Since LPA is trained on adjacent MOVi-F frames, how does its performance vary with increasing temporal gaps, large motion, or prolonged occlusion during evaluation?

Answer: LPA-CWM remains applicable beyond adjacent frames, with accuracy decreasing as temporal gaps, motion, and occlusion duration increase.

We evaluate LPA-CWM on DAVIS under three conditions: query-to-target temporal gaps of 1, 5, 10, 20, and 40 frames; ground-truth displacements of 0–4, 4–16, and 16–64 pixels at a fixed five-frame gap; and continuous occlusions lasting 1–15 or 16–50 frames. The MOVi-F-trained checkpoint remains frozen, without fine-tuning. We report core TAP-Vid and CMC metrics for these conditions below.

ConditionSettingAJ ↑⟨δavg⟩ ↑AD ↓Coverage ↑DCA_avg ↑VisBA ↑OF1 ↑
Temporal gap1 frame72.2075.781.92100.0075.16100.00100.00
Temporal gap5 frames53.8664.583.34100.0063.3788.1579.63
Temporal gap10 frames44.8659.839.8596.9157.5376.7659.14
Temporal gap20 frames41.7950.7113.5995.2449.8875.9649.19
Temporal gap40 frames25.0737.1829.4174.6031.1169.4035.47
Motion at 5-frame gap0–4 px72.5673.631.95N/A75.18N/AN/A
Motion at 5-frame gap4–16 px63.6966.672.73N/A66.67N/AN/A
Motion at 5-frame gap16–64 px53.7155.945.21N/A55.94N/AN/A
Occlusion duration1–15 frames23.6744.8611.2682.2233.9154.1733.97
Occlusion duration16–50 frames9.3517.1743.7680.8914.7040.0012.50

Motion subsets contain only ground-truth-visible targets; their non-informative coverage/visibility entries are omitted as N/A.

LPA-CWM performs best on adjacent frames and retains correspondence accuracy across 5–20-frame gaps (AJ 53.86–41.79) until a more pronounced decline occurs at 40 frames (AJ 25.07). At a fixed five-frame gap, accuracy decreases as displacement increases, but the 16–64-pixel group still achieves AJ 53.71 with AD 5.21 pixels. Prolonged occlusion is more challenging: localization accuracy decreases, although average within-threshold accuracy remains 17.17% after 16–50 occluded frames. These results show that the LPA module remains applicable beyond its training setting.

  • Question 2: How much of the final improvement comes from learned candidate weighting versus the subsequent CWM re-evaluation? A matched comparison across different re-evaluation budgets would help clarify their respective contributions.

Answer: Learned candidate weighting reduces EPE by 0.4172 px without re-evaluation, and one subsequent CWM re-evaluation further reduces EPE by 0.5660 px. We fix M=10 and compare Uniform-CWM with LPA-CWM at R=0, then compare LPA-CWM at R=0 and R=1. The LPA-CWM budget results are also reported in Fig. E.1.

MethodMREPE ↓
Uniform-CWM1003.1195
LPA-CWM1002.7023
LPA-CWM1012.1363

The R=0 comparison isolates the gain from learned weighting without re-evaluation; the R=0→1 comparison measures the further gain from re-evaluation within LPA-CWM.

AuthorsOpenReview ↗

Response to Reviewer GHnp (part 2)

Thank you for the constructive feedback. We address the questions and weaknesses below.

  • Question 3: What is the additional inference overhead of LPA compared with Uniform CWM and APA-CWM under identical settings? How does the method scale with the number of candidates and tracked points?

Answer: LPA adds 0.1844% inference overhead in the reported evaluation subset, and its overhead ranges from 0.32% to 0.61% when tracking 1–16 points. We measured Opt-CWM, Opt-CWM + APA, and Opt-CWM + LPA under identical settings, and evaluated 1, 8, and 16 simultaneous queries on a fixed frame pair.

Mean inference overhead during evaluation

MethodMean time (ms/point–frame request)Additional time (ms/point–frame request)Overhead
Opt-CWM3393.4738——
Opt-CWM + APA3418.6650+25.1912+0.7423%
Opt-CWM + LPA3399.7326+6.2587+0.1844%

Scaling with tracked points

Tracked points QMethodMean time (ms/call)Additional time (ms/call)Overhead
1Opt-CWM4949.658——
1Opt-CWM + APA4988.637+38.979+0.7875%
1Opt-CWM + LPA4980.036+30.378+0.6137%
8Opt-CWM27181.519——
8Opt-CWM + APA27420.951+239.433+0.8809%
8Opt-CWM + LPA27283.016+101.497+0.3734%
16Opt-CWM53061.838——
16Opt-CWM + APA53875.118+813.280+1.5327%
16Opt-CWM + LPA53230.759+168.920+0.3183%

Scaling with candidate count

We reproduce the R=1 results from Fig. E.1 below: the original Uniform-CWM-based LPA-CWM is evaluated on 40 fixed MOVi-F query groups, with latency measured on an RTX 5090. These are separate from the complete Opt-CWM timings above.

Candidates MRe-evaluations REPE (px) ↓Mean time (ms/query) ↓
512.54601365
1012.13632716
2012.37875400

LPA has only 3.0M parameters and weights the initial candidates once, while the underlying CWM requires multiple expensive forward passes. Runtime increases with query and candidate counts, but LPA's added cost remains small relative to Opt-CWM. M=10 provides the best accuracy among these settings, while M=20 costs more without improving accuracy.

  • Question 4: LPA is supervised by endpoint errors and relies primarily on visual and response features, without explicitly modeling physical constraints. It remains unclear whether the learned weights capture physical knowledge or merely correspondence confidence.

Answer: We agree that LPA learns candidate reliability from visual and response features under endpoint-error supervision; the physically informed starting point of our framework is the analytic APA module.

(1) LPA learns the relative reliability of motion candidates. As the reviewer observes, our supervision assigns higher target weights to candidates with smaller distances to the ground-truth endpoint. LPA uses visual evidence, candidate response features, and motion information to identify more reliable candidates; its learned weights reflect correspondence reliability rather than specific physical knowledge.

(2) Our method builds on APA, an analytic module incorporating a physics-inspired prior. The prior used by APA draws on NSG, which relates spatial structure to temporal changes through a probability-mass continuity equation, analogous to mass conservation in fluid mechanics. APA uses this spatiotemporal consistency prior to assign analytic candidate weights, providing the starting point for our learnable aggregation approach.

Our new DAVIS experiments apply APA and the existing LPA checkpoint to complete Opt-CWM without additional training or fine-tuning, under the same evaluation settings:

TAP-Vid results

MethodAJ ↑⟨δ_avg⟩ ↑AD ↓MD ↓OA ↑
Opt-CWM45.6162.1010.323.0680.21
Opt-CWM + APA46.5462.829.882.8881.32
Opt-CWM + LPA47.0163.548.572.7881.69

CMC results

MethodCov. ↑[email protected] ↑DCA ↑CTS ↑VisBA ↑OF1 ↑LCC ↑
Opt-CWM83.7572.8056.8349.7771.9849.5677.91
Opt-CWM + APA84.5675.7157.5352.0074.8552.5178.92
Opt-CWM + LPA85.2976.0958.1752.3474.8952.6779.99

AD and MD are in pixels; other values are percentages.

The analytic APA module incorporating this physics-inspired prior already improves performance across the reported measures. Building on this aggregation framework, LPA provides a learnable reliability estimator that further improves performance.

AuthorsOpenReview ↗

Response to Reviewer GHnp (part 3)

Thank you for the constructive feedback. We address the questions and weaknesses below.

  • Question 5: Although the ablations validate individual components, the paper does not compare against simpler learned reliability estimators, such as an MLP-based candidate scorer. This makes it difficult to assess whether the proposed architecture is necessary for the reported gains.

Answer: The “w/o candidate-set interaction” ablation in our original Table 2 is an MLP-based candidate scorer, and the full LPA architecture improves performance beyond this simpler learned baseline. We apologize that, due to space constraints, the detailed settings were placed in the appendix and this row was not explicitly labeled as an MLP scorer.

This variant retains the same input branches and feature-encoder design for local visual evidence, candidate responses, frame-pair context, and scalar statistics. It removes the Set Transformer and applies a shared MLP independently to each candidate token, followed by softmax across candidates. It is separately trained with the same MOVi-F training data and the same complete three-term training loss. All variants use the same experimental configuration (Appendix E).

MethodAJ ↑AD ↓DCA ↑CTS ↑LCC ↑
Uniform-CWM24.3113.1036.8721.4685.20
MLP scorer (w/o candidate-set interaction)41.528.9951.8645.5590.30
LPA-CWM45.296.7358.9857.5493.18

AD is in pixels; other values are percentages.

The MLP scorer already improves over uniform weighting, while adding candidate-set interaction further improves all reported measures: AJ increases by 3.77 percentage points and DCA increases by 7.12 percentage points relative to the MLP baseline. These controlled results demonstrate the clear contribution of the proposed architecture to the reported performance gains.

  • Question 6: It remains unclear whether the learned weights capture generalizable counterfactual evidence or protocol-specific response patterns.

Answer: The same frozen LPA checkpoint improves complete Opt-CWM without fine-tuning, supporting transferable counterfactual motion evidence across the tested CWM inference configurations.

Our original Uniform-CWM and the complete Opt-CWM differ in both input resolution and multiscale refinement:

SettingOriginal Uniform-CWMComplete Opt-CWM
Input resolution256 × 256512 × 512
Official multiscale refinementNot enabledMS=4

We transfer the existing LPA checkpoint directly to complete Opt-CWM on DAVIS, without retraining or fine-tuning, while keeping the underlying CWM frozen. Reviewer kdjQ also highlights the generalization of the MOVi-F-trained module across diverse real-world datasets without dataset-specific adaptation; our new Opt-CWM experiment additionally tests transfer across inference configurations. Under the same complete Opt-CWM settings, adding LPA produces the following gains:

MethodAJ ↑DCA ↑CTS ↑LCC ↑
Opt-CWM45.6156.8349.7777.91
Opt-CWM + APA46.5457.5352.0078.92
Opt-CWM + LPA47.0158.1752.3479.99

All values are percentages.

The gains persist after changing resolution and enabling multiscale refinement, despite reusing the same learned module. These results support that LPA captures transferable counterfactual motion evidence, with its candidate-reliability estimates remaining useful beyond the original response configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.