acceptodds
Under review as a conference paper at ICLR 2027

Your "Clean Split" May Not Be Clean: Measuring Recording-Level Leakage in Video Segmentation Benchmarks

Abstract

Video-derived segmentation benchmarks are routinely split at frame or clip level, so near-copies of one recording fall on both sides of the split. That this leaks information is known; we ask whether the standard way of measuring it is itself reliable. The conventional comparison subtracts scores from different test sets, confounding the memorisation gain with test-set difficulty. We propose two tools. Protocol-matched evaluation re-scores both protocols' checkpoints on a common test set, separating the test-set swap from the contrast between the two training protocols without retraining: on infrared, visible and a second public benchmark the matched gain is +6.4 to +8.8 IoU, of which the conventional comparison captures 36%, 68% and −17%. Near-duplicate auditing against adjacent-frame and random-pair references tests whether a "clean" split is clean: that benchmark's recording-disjoint split is as duplicated as adjacent frames (72.5% against 70.6%), which is consistent with why the conventional measurement fails there, while the primary benchmark's split passes on both bands (0.5% visible, 6.9% infrared). On the common clean test set the visible-band night score rises from 0.73 to 0.91, and comparisons on the protocols' respective test sets reverse the observed modality ordering. We release the audit and protocol-matched evaluation code with every per-architecture result.

Primary area:
datasets and benchmarks
Keywords:
data leakage, benchmark integrity, evaluation protocol, video segmentation, near-duplicate detection, train-test contamination, recording-level cross-validation, anti-UAV

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Tx4DOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of data leakage in video segmentation tasks. It introduces a near-duplicate auditing method to quantify data leakage and analyzes the extent of leakage under different data-splitting strategies. Furthermore, it proposes a protocol-matched evaluation approach that evaluates models trained under different data-splitting protocols on the same clean test set, aiming to provide a more reliable assessment of the performance differences between models trained with and without data leakage.

Critical strengths
  1. Data leakage is an important issue in computer vision tasks. This paper provides a comprehensive analysis of both data splitting strategies and evaluation protocols, offering valuable insights for downstream applications.

  2. The supplementary materials provide extensive code and supporting data, enhancing the reproducibility and credibility of the experimental results.

Critical weaknesses
  1. Quantifying data leakage is an important issue that has been extensively studied in machine learning [1]. This paper proposes a near-duplicate auditing method to assess the extent of data leakage. However, prior work [1] has explored a similar approach based on detecting near-duplicate samples, which raises concerns about the novelty of the proposed method.

  2. It is somewhat unclear why the clean test set is considered a more appropriate evaluation set. Although it has little or no overlap with the clean training set, it may still substantially overlap with the leaky training set. Could the authors clarify why evaluating both models on the clean test set provides a more reliable assessment of data leakage?

[1] Do we train on test data? Purging CIFAR of near-duplicates. Journal of Imaging, 2020.

Questions for authors

Please refer to the Weaknesses section for details.

Reviewer_aBKEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies data leakage in video segmentation. It compares models on the same test set. It also checks for similar frames across training and test sets.

Critical strengths

Using the same test set avoids differences in test difficulty. The gain is positive in all 15 tested settings.

Critical weaknesses
  1. The comparison does not separate leakage from other changes. The leaky model sees more recordings. Its validation set is also contaminated. It trains on 35 of the 50 clips in the shared test set (Section 4.5 and Appendix A.6). These changes occur together. The authors acknowledge this but do not run a control to separate their effects.
  2. The similarity threshold is not well tested. The audit compares each test frame with all training frames and takes the highest similarity. The reference check uses only single pairs. These are different tests. The paper does not measure the false-positive rate for the full search. The authors also chose 0.99 after running the audit (Appendix A.2).
  3. The CST result has no checked examples. The paper reports a 72.5% near-copy rate. It does not show or inspect the CST frame pairs behind this number. Figure 1 illustrates how videos are split. In Anti-UAV, the authors found that nearly blank skies can produce high similarity (Appendix A.2). They need to check whether CST has the same problem.
  4. The main score gives some recordings more weight. Table 1 pools IoU across frames. Recordings with a larger total foreground union count more. The Anti-UAV test set has only six recordings. Only two are at night. The separate cross-validation results do not show how stable the gains in Table 1 are. The paper should report the gain for each recording and repeat the data split.
  5. The writing and organization are hard to follow. The same limitations appear in Section 4.5, Section 8 and Appendix A.6. The title of Appendix A.9 refers to the Anti-UAV split. Its text instead describes CST and still-image datasets. The appendix tables are dense. Frequent cross-references make the reader jump between sections.
Questions for authors
  1. Can you match the number of training recordings and use an independent validation set? Does the gain remain?
  2. Can you show checked CST frame pairs? Can you test the threshold on unrelated data using the same search procedure and pool size?
  3. Do the results hold when each recording has equal weight? Do they hold across new splits?
Reviewer_JewPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines recording-level leakage in video-derived segmentation benchmarks and how it can be measured. It argues that comparing a leaky-trained model on its corresponding test split with a clean-trained model on its own test split is confounded: the score difference reflects both the training protocol and the choice of test data. The paper therefore proposes protocol-matched evaluation, in which both models are evaluated on a common clean test split. Additionally, it audits train–test similarity in nominally recording-disjoint partitions using downsampled grayscale correlation, with adjacent-frame and unrelated-pair similarities as references. Experiments on paired visible and infrared Anti-UAV data and CST Anti-UAV yield larger mean common-test contrasts than conventional comparisons, showing that changing the test set can substantially obscure or even reverse the observed difference between training protocols. These results highlight the need to compare models on the same test set and to audit recording-disjoint splits for near-duplicates before treating them as clean.

Critical strengths
  1. The paper provides a useful demonstration of how test set changes can alter a protocol comparison. It shows that evaluating two models on their respective test sets can greatly obscure their observed difference. The conventional versus fixed-test protocol contrast means are +2.3 versus +6.4 points on Anti-UAV infrared, +5.7 versus +8.5 on visible, and −1.5 versus +8.8 on CST. The evaluation is easy to apply when the checkpoints are available and provides an interpretable diagnostic.
  2. The paper additionally checks whether recording-disjoint training and test sets still contain highly similar images across the split. The audit flags the CST split, where the cross-split near-copy rate is comparable to the rate for adjacent frames, suggesting potential train–test contamination.
Critical weaknesses
  1. The robustness and interpretation of the reported gains need stronger support. The main protocol-matched experiments use one pair of clean and leaky splits for Anti-UAV and one for CST. The two Anti-UAV imaging bands share the same recording partition, and its clean test contains only six recordings. Additional training seeds are reported for some configurations, but the main comparison is not repeated across independently generated data partitions. The five-fold cross-validation concerns architecture comparisons rather than the protocol-matched gains. It therefore remains unclear how sensitive the reported gains and the CST sign reversal are to the chosen splits. Furthermore, as the authors acknowledge, the protocols differ in recording coverage and checkpoint selection as well as exposure to test data. The score difference therefore cannot be attributed to test-data leakage alone, which limits the interpretation of the reported “fraction of leakage captured.”
  2. The CST audit result needs more direct validation. High correlation between small grayscale images may reflect similar backgrounds rather than repeated scene content. The authors inspected this ambiguity on Anti-UAV but did not inspect the matched CST pairs. Therefore, the reported 72.5% rate indicates high visual similarity, but does not show how many matches represent repeated content or whether they inflate segmentation scores. Furthermore, the audit selects the closest match from many training images, while its references compare individual pairs. These references do not establish how often the search produces false matches. Inspecting CST pairs and validating the search against suitable controls would strengthen the conclusion.
Questions for authors
  1. Could you repeat the protocol-matched evaluation with different clean and leaky splits, at least for one representative architecture? This would show whether the matched gains and the sign reversal on CST are stable across data splits.
  2. Could you report the matched gain separately for the clean-test clips that belong to leaky train, leaky validation, and leaky test (35/6/9 clips on Anti-UAV)? This would show whether the improvement is concentrated on frames used directly for training or also occurs on other frames from the same recordings.
  3. The cross-split score takes the maximum similarity over the training set, while the random-pair null uses individual pairs. Could you add an experiment in which each frame is compared against 1,540 candidate frames from known unrelated recordings, retaining the maximum similarity? Comparing the resulting near-copy rate with the CST rate would help assess accidental matches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.