acceptodds
Under review as a conference paper at ICLR 2027

PILOT: Self-Supervised Visual Representation Pretraining for Temporal Correspondence

Abstract

Self-distillation has emerged as a strong paradigm for self-supervised visual representation learning. However, extending this approach to learning video representations with strong temporal correspondence remains challenging: raw frames provide no explicit patch-to-patch alignment over time, while self-supervised pretraining must derive correspondence supervision without external optical flow, point tracks, or correspondence annotations. We address this bottleneck with PILOT (Privileged Information Learning via Optimal Transport), a self-supervised representation pretraining framework based on explicit cross-frame target construction. PILOT uses asymmetric distillation: a privileged teacher combines information privilege with algorithmic privilege (Sinkhorn-Knopp optimal transport) to produce globally constrained cross-frame patch assignment predictions that serve as targets for a restricted student. These targets exhibit reduced assignment imbalance and many-to-one collisions, while the student predicts them from masked inputs using row-wise Softmax, encouraging the student encoder to internalize cross-frame correspondence. After pretraining, only a standard encoder is retained. Experiments show that PILOT achieves state-of-the-art overall performance among self-supervised pretrained encoders across three dense correspondence benchmarks and demonstrates strong transfer to action recognition and video retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Pretraining, Video Representation Learning, Dense Temporal Correspondence, Vision Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oZyyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on how to learn visual features with reliable cross-frame correspondence through self-supervised pretraining without explicit temporal correspondence supervision. It uses a teacher-student architecture and a masking strategy. The teacher uses multiple unmasked frames to generate patch matching relationships between different frames in a video. The student uses the masked current frame to predict the patch correspondence distribution between the current frame and the teacher's reference-frame features. The experiments use common video datasets and evaluation metrics for tasks such as action recognition and segmentation, and the results show slight overall improvements.

Critical strengths

1.The proposed teacher-student framework directly addresses the temporal correspondence problem. Using an unmasked teacher to generate cross-frame matching targets provides a reasonable way to guide the student without requiring correspondence annotations. 2.The experimental evaluation is relatively comprehensive, covering several dense correspondence benchmarks and video-level tasks. The ablation studies and occlusion experiments also help demonstrate the effectiveness of the proposed method.

Critical weaknesses

1.Sinkhorn may produce incorrect matches when objects are fully occluded or leave the frame. Although the paper evaluates occlusion robustness, these cases are not directly analyzed. 2.The method mainly combines existing techniques, including teacher-student distillation, Sinkhorn routing, and feature perturbation, so its novelty is somewhat limited. The main contribution lies in combining these components rather than introducing a fundamentally new technique. In addition, the improvements on video-level tasks are not consistent across all benchmarks. For example, PILOT does not achieve the best results on HMDB51 action recognition or SSv2, which raises some concerns about its generalization to broader video understanding tasks.

Questions for authors

1.How does the method handle patches without valid correspondences, such as objects leaving the frame or becoming fully occluded? Could these incorrect matches affect student training? 2.How does PILOT perform when the temporal gap between frames increases or objects undergo large motions? 3. PILOT performs well on dense correspondence tasks, but its improvements on video-level tasks are less consistent. Why does it not outperform existing methods on HMDB51 and SSv2? 4. The paper shows that Sinkhorn produces more balanced patch assignments, but does this necessarily lead to more accurate correspondences? Could the authors provide a direct evaluation of the teacher-generated matching targets?

5 comments

We thank the reviewer for their time and constructive feedback. We have addressed all the concerns through additional experiments and clarifications. The revised manuscript incorporates these results with changes highlighted in dark red. We provide our responses below.

1. W1/Q1: Patches without valid correspondences

We have already investigated patches without valid correspondences through a dustbin ablation in the original submission (Appendix C.9). We agree that this issue should be explained more clearly and have now included the ablation results in the main text (Sec. 4.3).

  • Dustbin ablation. We append an additional dustbin column to the teacher similarity matrix, with each row receiving a fixed or learnable matching score for the unmatched option. During Sinkhorn normalization, this option participates in the assignment alongside reference patches, allowing current patches to assign probability to the dustbin. All three dustbin variants underperform the default formulation, which achieves better downstream performance while maintaining a simpler design without additional dustbin hyperparameters.

    DustbinDustbin ScoreJ&F ↑J ↑F ↑
    Fixed score0.165.863.767.9
    Fixed score0.665.363.667.0
    Learnable scoreLearned scalar64.662.866.4
    Without dustbin (default)—66.764.868.6
  • Analysis of unmatched patches. We examine all DAVIS-2017 validation videos and identify 2008 frame pairs with naturally unmatched object instances, where the current-frame object is completely absent from the reference frame. For these unmatched foreground patches, we measure the peak teacher assignment probability and normalized row entropy, comparing Softmax and Sinkhorn using identical teacher similarity matrices.

    Sinkhorn produces much lower peak probabilities and higher entropy than Softmax, indicating less concentrated teacher assignments when no valid object-level match exists. While PILOT has no explicit unmatched option, these assignments reduce reliance on any single potentially incorrect match. The three tested dustbin variants do not improve downstream performance, suggesting that the default soft-assignment formulation provides a more effective empirical tradeoff in our setting.

    Teacher RoutingMean Peak Probability ↓Normalized Row Entropy ↑
    Row-wise Softmax8.3%0.87
    Sinkhorn (Ours)1.5%0.98

2. Q2: Larger temporal gaps and object motions

We added an evaluation on DAVIS-2017 by sampling every 10th, 20th, and 40th frame from the original sequences to assess correspondence over longer temporal intervals. PILOT consistently outperforms all baselines, with its J&Fₘ gain over the strongest baseline increasing from +2.7 pp at stride 10 to +3.9 pp at stride 40.

Meanwhile, as the sampling stride increases from 10 to 40 frames, the median object-center displacement rises from 3.7% to 11.2% of the image diagonal, indicating larger object motions at longer temporal intervals. These results demonstrate that PILOT maintains stronger correspondence performance under increasingly sparse temporal sampling and larger observed object motions. Values in parentheses indicate gains (pp) over the strongest baseline.

Method10-frame J&F10-frame J10-frame F20-frame J&F20-frame J20-frame F40-frame J&F40-frame J40-frame F
RSP52.550.854.348.647.350.043.041.244.8
iBOT56.755.358.253.051.155.045.143.047.2
DINOv256.255.057.452.650.155.144.542.646.4
T-CoRe56.555.457.553.251.654.945.544.246.7
PILOT59.4 (+2.7)57.3 (+1.9)61.4 (+3.2)56.2 (+3.0)53.8 (+2.2)58.5 (+3.4)49.4 (+3.9)47.2 (+3.0)51.5 (+4.3)

3. W2: Methodological novelty

PILOT's main contribution is not the individual use of existing components, but formulating explicit cross-frame patch assignment prediction as a novel self-supervised pretraining objective, where a privileged teacher constructs globally constrained targets to supervise a masked student.

Unlike approaches relying on reconstruction, temporal prediction, or prototype-based feature learning, PILOT directly distills dense cross-frame correspondence targets to learn visual representations with strong temporal correspondence, without external optical flow, point tracks, or correspondence annotations. Reference perturbation further mitigates positional shortcuts in target construction.

4. W2/Q3: Transfer to video-level tasks

PILOT is designed for dense temporal correspondence, and we additionally evaluate action recognition and video retrieval to assess the transferability of the learned representations. As reported in the submission, PILOT outperforms T-CoRe, the prior leading method for self-supervised temporal correspondence learning, across all five evaluations and achieves the best results among all methods in three. On HMDB51 and SSv2 action recognition, it trails the best methods by only 0.4 and 0.6 pp, respectively. Importantly, PILOT leads on both video retrieval benchmarks, with an average gain of +3.3 pp over the strongest baselines. Unlike action recognition, which requires a trained linear classification head, video retrieval directly compares frozen video representations through cosine similarity without additional training. These results provide complementary evidence of representation transfer beyond dense correspondence.

To better understand the small performance gaps on HMDB51 and SSv2 action recognition, we note that both datasets contain classes recognizable from individual frames (e.g., smiling in HMDB51 and holding something in SSv2), whereas PILOT emphasizes temporal correspondence learning. We therefore evaluate two temporally focused subsets identified in prior work: the 10-class HMDB51 motion-focused subset from SimVA (So et al.), which selects actions difficult to recognize from a single frame, and the 18-class temporal SSv2 subset based on Sevilla-Lara et al., which emphasizes temporal ordering. PILOT achieves the best accuracy on both (38.4% and 36.4%), outperforming the strongest baselines by +2.1 and +2.5 pp, respectively. These results show a more pronounced advantage on temporally focused subsets, despite the small gaps on the full benchmarks.

MethodSimVA (HMDB51)Temporal-SSv2
RSP26.326.2
iBOT35.032.8
DINOv233.833.9
T-CoRe36.333.6
PILOT38.4 (+2.1)36.4 (+2.5)

5. Q4: Direct correctness of teacher targets

We added a direct evaluation of teacher-generated matching targets against DAVIS-2017 ground-truth instance masks. For each evaluated foreground patch in the current frame, a top-1 match is counted as correct when the current patch and its selected reference patch belong to the same annotated object instance. All four routing methods use identical teacher similarity matrices and reference perturbation, changing only the routing operator.

Sinkhorn achieves 75.9% top-1 same-instance accuracy, outperforming Softmax by +21.2 pp, Dual-Softmax by +13.1 pp, and Column Centering by +9.0 pp. These results directly demonstrate improved object-instance consistency of the teacher's matching targets, beyond the assignment balance and collision reductions reported in the original submission.

Teacher RoutingTop-1 Instance Accuracy ↑
Row-wise Softmax54.7%
Dual-Softmax62.8%
Column Centering66.9%
Sinkhorn (Ours)75.9%

Dear Reviewer,

Thank you again for your thoughtful feedback. We understand this is a busy period and sincerely appreciate your time.

We would be grateful if you could review our rebuttal and let us know whether it addresses your concerns. We are happy to provide any further clarification.

Best regards, The Authors

Reviewer_oLWFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PILOT pretrains a frame-level vision transformer for dense temporal correspondence without optical flow, point tracks, or correspondence annotations. An EMA teacher processes unmasked current and reference frames, performs parameter-free cross-frame fusion, perturbs the reference features, and applies Sinkhorn–Knopp routing to construct patch-assignment targets. A masked student predicts these targets using row-wise Softmax, jointly with an iBOT-style masked-image-modeling objective. The default ViT-S/16 model is pretrained on Kinetics-400 using 350 MIM epochs followed by 50 joint-objective epochs. Evaluation covers frozen label propagation on DAVIS-2017, VIP, and JHMDB, as well as linear action recognition, video retrieval, ablations, run-to-run variation, and computational cost.

Critical strengths
  1. PILOT formulates cross-frame matching as an explicit distillation target while retaining only a standard encoder after pretraining, introducing no inference-time OT dependency.

  2. With ViT-S/16, PILOT improves over the listed T-CoRe result by 2.0 points on DAVIS-2017 and 2.3 points on VIP, while obtaining the best listed [email protected] on JHMDB (Table 1). ViT-B results preserve the DAVIS and VIP gains (Table 9).

Critical weaknesses
  1. The evidence does not establish that Sinkhorn produces more accurate correspondence targets or genuinely improves relative occlusion robustness.According to Section 3.2 and Eq. (4); Tables 2, 4, and 5; Appendix B, Eqs. (17)–(19); Table 17, lower column deviation and fewer collisions are largely consequences of Sinkhorn’s column constraints, not measurements of correspondence correctness. Uniform marginals also force every reference patch to receive mass despite occlusion, disocclusion, scale changes, and genuinely unmatched regions. Under synthetic occlusion, PILOT has the highest absolute accuracy, but its drops from the clean score are 10.9, 17.8, and 21.8 points, compared with 9.6, 17.3, and 21.2 for T-CoRe. Thus Table 5 demonstrates higher occluded-input accuracy, not smaller relative degradation. The natural-occlusion and controlled routing results are suggestive but lack subset counts and uncertainty.
Questions for authors
  1. What exact Sinkhorn updates produce QQ, how does ϵ\epsilon relate to τ\tau, and how are the marginals defined for the N∗(N+1)N *(N+1) dustbin variants?

  2. Are there separate gθg_\theta and EMA gξg_\xi heads? Which tensors are detached, what EMA schedule is used, and is LcorrL_{\mathrm{corr}} applied to all patches or only masked patches?

  3. Which DAVIS sequences belong to the occlusion and non-occlusion subsets, how many samples does each contain, and what are the paired Softmax-versus-Sinkhorn results on both subsets with uncertainty estimates?

  4. What checkpoint, input resolution, feature layer, preprocessing, dataset split, and model-selection conventions underlie the results in Tables 1 and 3?

5 comments

We thank the reviewer for the time and effort in providing constructive reviews. We have addressed all the concerns through additional experiments and clarifications. The revised manuscript includes corresponding updates, highlighted in dark red. Our responses are provided below.

1. W1: Teacher target correctness

To directly assess teacher target correctness, we evaluate the predicted correspondences against DAVIS-2017 ground-truth instance masks. All four routing methods use identical teacher similarity matrices and reference perturbation, changing only the routing operator. A top-1 match is considered correct when the current patch and its selected reference patch belong to the same annotated object instance.

Sinkhorn achieves 75.9% top-1 instance accuracy, outperforming Softmax by +21.2 pp, Dual-Softmax by +13.1 pp, and Column Centering by +9.0 pp. This provides direct evidence of improved target correctness beyond assignment balance and collision reductions.

Teacher routingTop-1 instance accuracy ↑
Row-wise Softmax54.7%
Dual-Softmax62.8%
Column Centering66.9%
Sinkhorn75.9%

To address the concern that uniform marginals may force incorrect assignments when an object is absent from the reference frame, we examine all DAVIS-2017 validation videos and identify 2008 frame pairs with naturally unmatched object instances, where the current-frame object is completely absent from the reference frame. Using identical teacher similarity matrices, Sinkhorn produces a lower mean peak assignment probability (1.5% vs. 8.3%) and higher normalized row entropy (0.98 vs. 0.87) than Softmax. These results indicate less concentrated teacher assignments when no valid object-level match exists, reducing reliance on any single potentially incorrect match.

We also investigated explicit unmatched handling by adding a dustbin column to the teacher similarity matrix, allowing patches without valid correspondences to assign probability to an unmatched option. As reported in the original submission (Appendix C.9), all three tested dustbin variants underperform the default no-dustbin formulation.

2. W1: Performance under synthetic occlusion

We thank the reviewer for raising this concern. PILOT achieves a +4.0 pp J&F gain over T-CoRe on the natural-occlusion subset and maintains the highest absolute J&F performance in the additional synthetic-masking stress test. To understand why its performance drop relative to clean inputs is larger than T-CoRe's, we separately evaluate unmasked and gray-masked object regions (at 20% masking). PILOT outperforms T-CoRe on unmasked regions (58.5 vs. 55.6 J&F), whereas T-CoRe performs better on gray-masked regions (35.9 vs. 32.1 J&F). Since gray pixels replace local object appearance while the original object labels must still be recovered, spatial proximity may provide useful matching cues in these regions.

  • Positional matching analysis. To investigate whether positional matching contributes to T-CoRe's advantage on gray-masked regions, we measure same-position top-1 matching on masked foreground patches. T-CoRe exhibits a higher same-position matching rate than PILOT (9.3% vs. 3.3%). For 89.2% of foreground patches, the patch at the same spatial index in the next frame belongs to the same GT object, showing why same-index matching can be useful over short temporal intervals.

    To further assess positional bias, we evaluate 300 frame pairs sampled from unrelated videos with different foreground objects and backgrounds (e.g., an aquarium goldfish vs. an indoor judo athlete). Despite these substantial visual differences, T-CoRe exhibits a much higher same-index matching rate than PILOT (26.3% vs. 9.1%), indicating a stronger positional preference.

    Together, these findings suggest that T-CoRe's stronger positional matching bias may contribute to its advantage on gray-masked regions and the smaller overall performance margin under synthetic masking.

    MethodUnmasked J&F ↑Gray-masked J&F ↑Masked Same-position@1Cross-video Same-position@1
    T-CoRe55.635.99.3%26.3%
    PILOT58.532.13.3%9.1%
  • Temporal gap analysis. Although matching patches at the same spatial position may seem advantageous, relying on this positional preference becomes a serious limitation as objects move across frames. To examine this, we evaluate temporal correspondence on original DAVIS-2017 frames using sampling strides of 10, 20, and 40. As the stride increases from 10 to 40, median object-center displacement rises from 3.7% to 11.2%, while the percentage of foreground patches whose same-position reference patch belongs to the same GT object drops from 53.4% to 31.3%. Meanwhile, PILOT's J&F gain over the strongest baseline increases from +2.7 to +3.9 pp. These results further support the benefit of reducing positional dependence under larger object motions.

    Frame strideMedian object center displacement (%)Same-location GT match (%)PILOT gain vs. best baseline (J&F)
    103.7%53.4%+2.7 pp
    206.1%39.3%+3.0 pp
    4011.2%31.3%+3.9 pp

Q1: Sinkhorn updates and dustbin marginals

Sinkhorn updates. We initialize L=SrtL=S_r^t, where SrtS_r^t is the temperature-scaled teacher similarity matrix defined in Eq. (3), with τ=0.1\tau=0.1. We perform three iterations in log space, each consisting of row normalization followed by column normalization:

L_{ij}\leftarrow L_{ij}-\log\sum_{i'}e^{L_{i'j}}.$$ We then apply a final row-wise normalization and obtain $Q=\exp(L)$, ensuring unit row sums and approximately balanced column marginals. **Relationship between $\epsilon$ and $\tau$.** In Eq. (17), $\epsilon=1$ since $S$ already incorporates $\tau$. Equivalently, the entropy coefficient is $\tau$ when using unscaled similarities. **Dustbin marginals.** For the $N\times(N+1)$ formulation, each row has unit mass, while real reference columns have capacity constraints (column sums at most 1). The dustbin absorbs unmatched mass without a fixed marginal. ### Q2: Projection heads, EMA, and loss coverage We use a single shared correspondence projection head (g), rather than separate student and EMA heads. Teacher features and Sinkhorn-generated targets are detached from gradient computation. The teacher encoder uses a cosine EMA momentum schedule from 0.996 toward 1.0. The correspondence loss ($L_{\mathrm{corr}}$) is applied to all current-frame patches, including both masked and visible patches. ### Q3: DAVIS subsets and uncertainty estimates Occlusion (8 sequences, 17 objects): bmx-trees, bike-packing, cows, horsejump-high, kite-surf, lab-coat, libby, parkour. Non-occlusion (22 sequences, 44 objects): blackswan, breakdance, camel, car-roundabout, car-shadow, dance-twirl, dog, dogs-jump, drift-chicane, drift-straight, goat, gold-fish, india, judo, loading, mbike-trick, motocross-jump, paragliding-launch, pigs, scooter-black, shooting, soapbox. We have also clarified the subset selection criteria in Appendix A.6 of the revised manuscript. We estimate paired 95% confidence intervals using 1000 video-level bootstrap replicates, resampling the same sequences for Softmax and Sinkhorn in each replicate. Sinkhorn improves both subsets, with a larger gain under occlusion. Both confidence intervals exclude zero. | Subset | Softmax | Sinkhorn | Gain (95% CI) | |:---|---:|---:|---:| | Occlusion | 54.3 | 58.5 | +4.2 [2.6, 7.3] | | Non-occlusion | 68.2 | 69.9 | +1.7 [0.9, 3.4] | ### Q4: Evaluation settings and model selection **Table 1 (dense correspondence).** Following SiamMAE, RSP, CropMAE, and T-CoRe, we use the same frozen-encoder label propagation protocol. For PILOT, we use the final 400-epoch EMA ViT-S/16 checkpoint and extract final-layer post-LayerNorm patch features. We evaluate the DAVIS-2017 validation set at 480×880, the VIP validation set with the long side resized to 880, and JHMDB test split 1 at 320×320, using ImageNet normalization. Cosine-similarity matching uses top-$k=7/10/7$ for DAVIS-2017, VIP, and JHMDB, respectively, with queue length 20 and neighborhood size 20. No downstream fine-tuning or checkpoint selection is performed (Appendix A). **Table 3 in the original submission (video-level transfer).** We use frozen pretrained ViT-S/16 encoders (same in Table 1) with final-layer post-LayerNorm patch features and uniformly sample eight frames per video, resized to 224×224 using bicubic interpolation. Patch features are spatially averaged within each frame and concatenated in temporal order. UCF101 and HMDB51 use official train/test split 1. Following VideoMAE (Tong et al., 2022), we train on the official SSv2 training split and report results on its validation split. For action recognition, we train linear classifiers for 100 epochs using Adam with a learning rate of 0.01 and report the final-epoch results for all methods. For video retrieval, each test video retrieves its nearest training set video using cosine similarity, without additional training (Appendix A).
Reviewer_fQbtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies self-supervised pretraining of a frame-level ViT encoder for dense temporal correspondence, without optical flow, point tracks, or correspondence annotations. The authors argue that the main difficulty is constructing reliable patch-level cross-frame targets from raw video, and propose PILOT, an asymmetric teacher-student distillation framework. An EMA teacher processes the unmasked current frame and a past or future reference frame, fuses them with a parameter-free cross-attention step, and applies Sinkhorn-Knopp to the patch similarity matrix to obtain an approximately doubly stochastic assignment target. A student that sees only the masked current frame predicts this target with row-wise Softmax, and the resulting correspondence loss is added to an iBOT-style masked image modeling loss; only the encoder is kept after pretraining. The authors also identify a positional shortcut that biases the Sinkhorn targets toward same-location matches, and mitigate it by adding cosine-decayed Gaussian noise to the teacher's reference features. With ViT-S/16 pretrained on Kinetics-400 (350 epochs of MIM followed by 50 epochs of PILOT), the paper reports the best overall label-propagation results among self-supervised encoders on DAVIS-2017, VIP, and JHMDB, with a larger gain on the DAVIS occlusion subset. It further provides video-level transfer experiments, component ablations, routing diagnostics, a ViT-B variant, repeated runs, and a cost analysis.

Critical strengths
  1. A well-targeted objective, with gains that hold up under a fair comparison. Instead of hoping that correspondence emerges from reconstruction, the paper directly supervises cross-frame patch assignments, and nothing extra (no flow, no tracker, no teacher) is needed at test time. The gains look real to me: with the same ViT-S/16 backbone, Kinetics-400 data, and 400-epoch budget, PILOT improves over T-CoRe by about 2 points on both DAVIS-2017 and VIP. I also appreciated the epoch-matched schedule control in Table 10 (simply splitting MIM into 350+50 epochs gives no gain), the three-run results (66.5 ± 0.3 on DAVIS), and the modest overhead (~13% more training time). This matters for my assessment because it shows the improvement is not coming from extra compute or a lucky run.
  2. An informative interaction between global routing and perturbation. Table 7(a) shows that Sinkhorn routing on its own actually hurts (62.8, below the 63.5 MIM baseline), and only becomes the best option once the reference features are perturbed (66.7). For me, this is the most transferable takeaway of the paper: anyone building OT-style targets on ViT features should know that a global assignment constraint can amplify positional bias. I think the explanation still needs more direct evidence (see weaknesses), but the finding itself is clearly documented and useful to the community.
Critical weaknesses
  1. The headline occlusion claim is not reliably supported. I could not reconcile Table 1 with Table 4. The full DAVIS score should be a weighted average of the two subsets, so PILOT's overall gain over T-CoRe (+2.0) should lie between its subset gains (+2.1 and +4.0), but it does not. Rounding cannot explain this unless the occlusion subset carries <5% of the weight, while the numbers imply roughly 19-28%. Perhaps the full-set baselines are taken from prior work, and the subsets were re-evaluated, but this is not stated. Also, under synthetic occlusion (Table 5), the margin over T-CoRe (+0.7 / +1.5 / +1.4) is smaller than on clean DAVIS (+2.0). Since occlusion robustness is a main claim (Fig. 1, contributions), this inconsistency weighs heavily in my rating.
  2. The mechanism is not isolated, and related work is incomplete. The balance and collision metrics in Table 2 largely follow from how Sinkhorn is defined; they do not show that the targets are more correct. A ground-truth check of target accuracy and a comparison with cheaper alternatives (e.g., dual-softmax) are missing. Also, a Sinkhorn teacher with a Softmax student is the core recipe of SwAV and DINOv2's centering, and Sinkhorn-based video SSL methods such as MoSiC, TimeT, DoRA, and SIGMA are not discussed. Without this, the contribution reads as an effective but mostly empirical combination of known parts. Both seem addressable, and I would reconsider my score if they are resolved.
Questions for authors
  1. Are the subset results in Table 4 from the same checkpoints and evaluation run as in Table 1? Please report full-set and subset scores for all methods under one protocol. Relatedly, how do the authors reconcile the smaller margins under synthetic occlusion (Table 5) with the occlusion claim?
  2. How accurate are the Softmax vs. Sinkhorn teacher targets against ground truth (e.g., top-1 matches falling inside the same object mask on DAVIS)?
  3. With the same perturbation, does Sinkhorn still outperform cheaper alternatives such as dual-softmax or column centering?
5 comments

We sincerely thank the reviewer for their time, effort, and constructive feedback. We have addressed all the concerns through additional experiments and clarifications. The corresponding changes are highlighted in dark red in the revised manuscript. We provide our responses below.

1. W1/Q1: Evaluation results in Tables 1 and 4

We thank the reviewer for raising the question about the consistency of the evaluation results in Tables 1 and 4. Table 1 uses baseline results reported in the original papers. Since these papers do not report occlusion/non-occlusion subset results, we re-evaluated their released checkpoints. To ensure a direct comparison, we now report full set and occlusion/non-occlusion subset results for all methods under a unified evaluation protocol in Table 4 of the revised manuscript. We have also clarified the sources of the reported results in the captions of both tables. All entries report J&F / J / F.

MethodOcclusionNon-occlusionFull Set
RSP50.1 / 47.1 / 53.162.5 / 61.6 / 63.459.1 / 57.6 / 60.6
iBOT54.3 / 51.3 / 57.367.0 / 65.4 / 68.663.5 / 61.5 / 65.5
DINOv254.1 / 51.1 / 57.265.8 / 64.8 / 66.962.6 / 61.0 / 64.2
T-CoRe54.5 / 51.4 / 57.567.8 / 66.8 / 68.764.1 / 62.6 / 65.6
PILOT58.5 / 54.7 / 62.369.9 / 68.7 / 71.166.7 / 64.8 / 68.6

2. W2/Q2: Teacher target correctness

Following the reviewer's suggestion, we further evaluate teacher target correctness against DAVIS-2017 ground-truth instance masks and compare Sinkhorn with simpler routing alternatives. For each foreground patch in the current frame, we select its top-1 reference patch and count the match as correct if both belong to the same annotated object instance. All four routing methods use identical teacher similarity matrices and perturbation.

Sinkhorn achieves 75.9% top-1 same-instance accuracy, outperforming Softmax by +21.2 pp, Dual-Softmax by +13.1 pp, and Column Centering by +9.0 pp. These results provide direct evidence that Sinkhorn improves teacher target correctness, beyond assignment balance and collision reductions.

Teacher RoutingTop-1 Same-Instance Accuracy ↑
Row-wise Softmax54.7%
Dual-Softmax62.8%
Column Centering66.9%
Sinkhorn (Ours)75.9%

3. Q3: Performance under different routing alternatives

Under the same pretraining protocol and reference perturbation, models pretrained with Dual-Softmax and Column Centering teacher targets achieve 65.1 and 65.3 J&F on DAVIS-2017, respectively, compared with 66.7 for Sinkhorn.

4. W1/Q1: Performance under synthetic occlusion

We thank the reviewer for raising this concern. Our occlusion claim is supported by PILOT's +4.0 pp J&F gain over T-CoRe on the natural-occlusion subset. PILOT also retains the best performance in the additional synthetic-masking stress test, with a smaller margin over T-CoRe than on clean DAVIS-2017. To examine this difference, we separately evaluate unmasked and gray-masked object regions (at 20% masking). PILOT outperforms T-CoRe on unmasked regions (58.5 vs. 55.6 J&F), whereas T-CoRe performs better on gray-masked regions (35.9 vs. 32.1 J&F). Since gray pixels replace local object appearance while the original object labels must still be recovered, spatial proximity may provide useful matching cues in these regions.

  • Positional matching analysis. To investigate whether positional matching contributes to T-CoRe's advantage on gray-masked regions, we measure same-position top-1 matching on masked foreground patches. T-CoRe exhibits a higher same-position matching rate than PILOT (9.3% vs. 3.3%). For 89.2% of foreground patches, the patch at the same spatial index in the next frame belongs to the same GT object, showing why same-index matching can be useful over short temporal intervals.

    To further assess positional bias, we evaluate 300 frame pairs sampled from unrelated videos with different foreground objects and backgrounds (e.g., an aquarium goldfish vs. an indoor judo athlete). Despite these substantial visual differences, T-CoRe exhibits a much higher same-index matching rate than PILOT (26.3% vs. 9.1%), indicating a stronger positional preference.

    Together, these findings suggest that T-CoRe's stronger positional matching bias may contribute to its advantage on gray-masked regions and the smaller overall performance margin under synthetic masking.

    MethodUnmasked J&F ↑Gray-masked J&F ↑Masked Same-position@1Cross-video Same-position@1
    T-CoRe55.635.99.3%26.3%
    PILOT58.532.13.3%9.1%
  • Temporal gap analysis. Although matching patches at the same spatial position may seem advantageous, relying on this positional preference becomes a serious limitation as objects move across frames. To examine this, we evaluate temporal correspondence on original DAVIS-2017 frames using sampling strides of 10, 20, and 40. As the stride increases from 10 to 40, median object-center displacement rises from 3.7% to 11.2%, while the percentage of foreground patches whose same-position reference patch belongs to the same GT object drops from 53.4% to 31.3%. Meanwhile, PILOT's J&F gain over the strongest baseline increases from +2.7 to +3.9 pp. These results further support the benefit of reducing positional dependence under larger object motions.

    Frame strideMedian object center displacement (%)Same-location GT match (%)PILOT gain vs. best baseline (J&F)
    103.7%53.4%+2.7 pp
    206.1%39.3%+3.0 pp
    4011.2%31.3%+3.9 pp

5. W2: Related Work and Methodological Novelty

We thank the reviewer for these references and have expanded the Related Work section in the revised manuscript to discuss these methods and clarify their differences from PILOT.

In image representation learning, SwAV uses Sinkhorn to generate balanced assignments of augmented image views to prototypes, while DINOv2 uses Sinkhorn to balance prototype assignments during image self-distillation. Both focus on learning image representations rather than temporal correspondences.

In video representation learning, TimeT uses Sinkhorn for patch-to-prototype clustering to learn temporally consistent features. SIGMA uses Sinkhorn to balance space-time tube assignments to prototypes for masked video modeling. DoRA uses Sinkhorn to associate object prototypes with patches within a frame for object discovery, while MoSiC uses Sinkhorn for feature-to-prototype clustering and propagates these assignments along externally tracked motion trajectories.

In contrast, PILOT shifts the pretraining focus toward explicit cross-frame patch-to-patch assignment prediction as a self-supervised objective, rather than prototype-based clustering. Sinkhorn operates directly on similarities between current-frame and reference-frame patches, enforcing global competition in constructing temporal correspondence targets. This requires no external optical flow estimators, point trackers, or correspondence annotations. We further show that global routing can amplify positional shortcuts, which reference feature perturbation effectively mitigates.

Dear Reviewer,

Thank you again for your constructive feedback. We sincerely appreciate your time and consideration.

We would be grateful if you could take a look at our rebuttal and let us know if any concerns remain. We would be happy to provide further clarification.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Summary

We thank the reviewers for their careful reading and constructive feedback.

In this work, we introduce a novel self-supervised pretraining objective to explicitly learn temporal correspondence: a privileged teacher generates globally constrained cross-frame patch assignment targets for a masked student to predict. We are encouraged that reviewers described the objective as “well-targeted” (fQbt), highlighted its use of an “explicit distillation target” (oLWF), and noted that the framework “directly addresses the temporal correspondence problem” (oZyy). In addition, we identify positional shortcuts that can affect globally constrained patch assignments and mitigate them through feature perturbation. Reviewer fQbt highlighted this interaction as “the most transferable takeaway of the paper.”

We also appreciate the positive assessment of our experiments, including “gains that hold up under a fair comparison” (fQbt) and a “relatively comprehensive” evaluation “covering several dense correspondence benchmarks and video-level tasks,” with ablation studies and occlusion experiments that “help demonstrate the effectiveness of the proposed method” (oZyy).

We have addressed each concern and question in our replies and highlighted manuscript revisions in dark red to help the AC and reviewers locate them. Below, we summarize our responses to the shared concerns.

1. Teacher target correctness (fQbt, oLWF, oZyy)

To assess whether balanced assignments lead to more accurate correspondences, we evaluated teacher-generated matches against DAVIS-2017 ground-truth instance masks, using identical teacher similarities and reference perturbation across all routing methods. Sinkhorn achieves 75.9% top-1 same-instance accuracy, compared with 54.7% for Softmax, 62.8% for Dual-Softmax, and 66.9% for Column Centering. These results provide direct evidence that Sinkhorn produces more accurate same-instance matches, beyond assignment balance.

2. Handling patches without valid correspondences (oLWF, oZyy)

  • Dustbin ablation. To handle patches whose corresponding objects are absent from the reference frame, we investigated adding an explicit unmatched option through a dustbin mechanism. This experiment was included in the appendix of the original submission and has now been added to the main text (Sec. 4.3). All three tested dustbin variants underperform the default no-dustbin formulation (64.6–65.8 vs. 66.7 J&F).
  • Analysis of routing behavior on naturally unmatched objects. We identified 2008 frame pairs where objects are completely absent from the reference frame. Compared with Softmax, Sinkhorn produces lower mean peak assignment probabilities (1.5% vs. 8.3%) and higher normalized row entropy (0.98 vs. 0.87). These results show that Sinkhorn produces less concentrated assignments when no valid match exists, reducing reliance on any single potentially incorrect match.

3. Performance under synthetic occlusion (fQbt, oLWF)

  • Synthetic occlusion and positional reliance. PILOT maintains the highest absolute accuracy under synthetic masking. To understand why its performance margin over T-CoRe is smaller than on clean DAVIS-2017, we investigated T-CoRe's reliance on positional matching. T-CoRe shows a stronger same-position matching preference than PILOT on gray-masked foreground patches (9.3% vs. 3.3%) and even between completely unrelated videos (26.3% vs. 9.1%). Under the standard evaluation setting (frame stride = 1), 89.2% of foreground patches have the same ground-truth object label as patches at the same spatial position in the next frame. In gray-masked regions, local object appearance is removed, but the original labels still must be recovered. This positional shortcut therefore gives T-CoRe an advantage in these regions, narrowing PILOT's overall performance lead under synthetic masking.
  • Larger temporal gaps. While same-position matching can act as a shortcut at short temporal gaps, it becomes less reliable under larger object motions. To examine this limitation, we evaluated temporal correspondence by sampling every 10th, 20th, and 40th frame from original DAVIS-2017 sequences. As the sampling interval increases, median object-center displacement rises from 3.7% to 11.2%, while same-position ground-truth matches decrease from 53.4% to 31.3%. Meanwhile, PILOT's gain over the strongest baseline increases from +2.7 to +3.9 pp, supporting the benefit of reducing positional dependence under larger object motions.

We sincerely thank all reviewers for their feedback. Reviewers oLWF and fQbt maintained their positive assessments, while Reviewer oZyy acknowledged that most concerns have been addressed and raised their rating from 2 to 3 (Weak Acceptance). All three reviewers now give positive ratings. We thank all reviewers and the AC again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.