- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the problem of data leakage in video segmentation tasks. It introduces a near-duplicate auditing method to quantify data leakage and analyzes the extent of leakage under different data-splitting strategies. Furthermore, it proposes a protocol-matched evaluation approach that evaluates models trained under different data-splitting protocols on the same clean test set, aiming to provide a more reliable assessment of the performance differences between models trained with and without data leakage.
Critical strengths
-
Data leakage is an important issue in computer vision tasks. This paper provides a comprehensive analysis of both data splitting strategies and evaluation protocols, offering valuable insights for downstream applications.
-
The supplementary materials provide extensive code and supporting data, enhancing the reproducibility and credibility of the experimental results.
Critical weaknesses
-
Quantifying data leakage is an important issue that has been extensively studied in machine learning [1]. This paper proposes a near-duplicate auditing method to assess the extent of data leakage. However, prior work [1] has explored a similar approach based on detecting near-duplicate samples, which raises concerns about the novelty of the proposed method.
-
It is somewhat unclear why the clean test set is considered a more appropriate evaluation set. Although it has little or no overlap with the clean training set, it may still substantially overlap with the leaky training set. Could the authors clarify why evaluating both models on the clean test set provides a more reliable assessment of data leakage?
[1] Do we train on test data? Purging CIFAR of near-duplicates. Journal of Imaging, 2020.
Questions for authors
Please refer to the Weaknesses section for details.