- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on how to learn visual features with reliable cross-frame correspondence through self-supervised pretraining without explicit temporal correspondence supervision. It uses a teacher-student architecture and a masking strategy. The teacher uses multiple unmasked frames to generate patch matching relationships between different frames in a video. The student uses the masked current frame to predict the patch correspondence distribution between the current frame and the teacher's reference-frame features. The experiments use common video datasets and evaluation metrics for tasks such as action recognition and segmentation, and the results show slight overall improvements.
Critical strengths
1.The proposed teacher-student framework directly addresses the temporal correspondence problem. Using an unmasked teacher to generate cross-frame matching targets provides a reasonable way to guide the student without requiring correspondence annotations. 2.The experimental evaluation is relatively comprehensive, covering several dense correspondence benchmarks and video-level tasks. The ablation studies and occlusion experiments also help demonstrate the effectiveness of the proposed method.
Critical weaknesses
1.Sinkhorn may produce incorrect matches when objects are fully occluded or leave the frame. Although the paper evaluates occlusion robustness, these cases are not directly analyzed. 2.The method mainly combines existing techniques, including teacher-student distillation, Sinkhorn routing, and feature perturbation, so its novelty is somewhat limited. The main contribution lies in combining these components rather than introducing a fundamentally new technique. In addition, the improvements on video-level tasks are not consistent across all benchmarks. For example, PILOT does not achieve the best results on HMDB51 action recognition or SSv2, which raises some concerns about its generalization to broader video understanding tasks.
Questions for authors
1.How does the method handle patches without valid correspondences, such as objects leaving the frame or becoming fully occluded? Could these incorrect matches affect student training? 2.How does PILOT perform when the temporal gap between frames increases or objects undergo large motions? 3. PILOT performs well on dense correspondence tasks, but its improvements on video-level tasks are less consistent. Why does it not outperform existing methods on HMDB51 and SSv2? 4. The paper shows that Sinkhorn produces more balanced patch assignments, but does this necessarily lead to more accurate correspondences? Could the authors provide a direct evaluation of the teacher-generated matching targets?
5 comments
4. W2/Q3: Transfer to video-level tasks
PILOT is designed for dense temporal correspondence, and we additionally evaluate action recognition and video retrieval to assess the transferability of the learned representations. As reported in the submission, PILOT outperforms T-CoRe, the prior leading method for self-supervised temporal correspondence learning, across all five evaluations and achieves the best results among all methods in three. On HMDB51 and SSv2 action recognition, it trails the best methods by only 0.4 and 0.6 pp, respectively. Importantly, PILOT leads on both video retrieval benchmarks, with an average gain of +3.3 pp over the strongest baselines. Unlike action recognition, which requires a trained linear classification head, video retrieval directly compares frozen video representations through cosine similarity without additional training. These results provide complementary evidence of representation transfer beyond dense correspondence.
To better understand the small performance gaps on HMDB51 and SSv2 action recognition, we note that both datasets contain classes recognizable from individual frames (e.g., smiling in HMDB51 and holding something in SSv2), whereas PILOT emphasizes temporal correspondence learning. We therefore evaluate two temporally focused subsets identified in prior work: the 10-class HMDB51 motion-focused subset from SimVA (So et al.), which selects actions difficult to recognize from a single frame, and the 18-class temporal SSv2 subset based on Sevilla-Lara et al., which emphasizes temporal ordering. PILOT achieves the best accuracy on both (38.4% and 36.4%), outperforming the strongest baselines by +2.1 and +2.5 pp, respectively. These results show a more pronounced advantage on temporally focused subsets, despite the small gaps on the full benchmarks.
| Method | SimVA (HMDB51) | Temporal-SSv2 |
|---|---|---|
| RSP | 26.3 | 26.2 |
| iBOT | 35.0 | 32.8 |
| DINOv2 | 33.8 | 33.9 |
| T-CoRe | 36.3 | 33.6 |
| PILOT | 38.4 (+2.1) | 36.4 (+2.5) |
5. Q4: Direct correctness of teacher targets
We added a direct evaluation of teacher-generated matching targets against DAVIS-2017 ground-truth instance masks. For each evaluated foreground patch in the current frame, a top-1 match is counted as correct when the current patch and its selected reference patch belong to the same annotated object instance. All four routing methods use identical teacher similarity matrices and reference perturbation, changing only the routing operator.
Sinkhorn achieves 75.9% top-1 same-instance accuracy, outperforming Softmax by +21.2 pp, Dual-Softmax by +13.1 pp, and Column Centering by +9.0 pp. These results directly demonstrate improved object-instance consistency of the teacher's matching targets, beyond the assignment balance and collision reductions reported in the original submission.
| Teacher Routing | Top-1 Instance Accuracy ↑ |
|---|---|
| Row-wise Softmax | 54.7% |
| Dual-Softmax | 62.8% |
| Column Centering | 66.9% |
| Sinkhorn (Ours) | 75.9% |
Dear Reviewer,
Thank you again for your thoughtful feedback. We understand this is a busy period and sincerely appreciate your time.
We would be grateful if you could review our rebuttal and let us know whether it addresses your concerns. We are happy to provide any further clarification.
Best regards, The Authors
We thank the reviewer for their time and constructive feedback. We have addressed all the concerns through additional experiments and clarifications. The revised manuscript incorporates these results with changes highlighted in dark red. We provide our responses below.
1. W1/Q1: Patches without valid correspondences
We have already investigated patches without valid correspondences through a dustbin ablation in the original submission (Appendix C.9). We agree that this issue should be explained more clearly and have now included the ablation results in the main text (Sec. 4.3).
Dustbin ablation. We append an additional dustbin column to the teacher similarity matrix, with each row receiving a fixed or learnable matching score for the unmatched option. During Sinkhorn normalization, this option participates in the assignment alongside reference patches, allowing current patches to assign probability to the dustbin. All three dustbin variants underperform the default formulation, which achieves better downstream performance while maintaining a simpler design without additional dustbin hyperparameters.
Analysis of unmatched patches. We examine all DAVIS-2017 validation videos and identify 2008 frame pairs with naturally unmatched object instances, where the current-frame object is completely absent from the reference frame. For these unmatched foreground patches, we measure the peak teacher assignment probability and normalized row entropy, comparing Softmax and Sinkhorn using identical teacher similarity matrices.
Sinkhorn produces much lower peak probabilities and higher entropy than Softmax, indicating less concentrated teacher assignments when no valid object-level match exists. While PILOT has no explicit unmatched option, these assignments reduce reliance on any single potentially incorrect match. The three tested dustbin variants do not improve downstream performance, suggesting that the default soft-assignment formulation provides a more effective empirical tradeoff in our setting.
2. Q2: Larger temporal gaps and object motions
We added an evaluation on DAVIS-2017 by sampling every 10th, 20th, and 40th frame from the original sequences to assess correspondence over longer temporal intervals. PILOT consistently outperforms all baselines, with its J&Fₘ gain over the strongest baseline increasing from +2.7 pp at stride 10 to +3.9 pp at stride 40.
Meanwhile, as the sampling stride increases from 10 to 40 frames, the median object-center displacement rises from 3.7% to 11.2% of the image diagonal, indicating larger object motions at longer temporal intervals. These results demonstrate that PILOT maintains stronger correspondence performance under increasingly sparse temporal sampling and larger observed object motions. Values in parentheses indicate gains (pp) over the strongest baseline.
3. W2: Methodological novelty
PILOT's main contribution is not the individual use of existing components, but formulating explicit cross-frame patch assignment prediction as a novel self-supervised pretraining objective, where a privileged teacher constructs globally constrained targets to supervise a masked student.
Unlike approaches relying on reconstruction, temporal prediction, or prototype-based feature learning, PILOT directly distills dense cross-frame correspondence targets to learn visual representations with strong temporal correspondence, without external optical flow, point tracks, or correspondence annotations. Reference perturbation further mitigates positional shortcuts in target construction.