- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReLaG, a scalable and modality-agnostic relation-aware data splitting framework. It addresses data leakage and overestimated generalization caused by related samples crossing train-test boundaries under random splitting. The method models sample dependence with a two-level latent-variable model, builds an approximate proximity graph with HNSW, infers related sample groups via Leiden community detection, and splits data at the community level as atomic units. It also introduces an adaptive threshold selection method based on unlabeled production data and the KS statistic, which can detect out-of-distribution data and estimate an effective dataset size. Experiments cover synthetic data, real molecular and protein datasets, and large-scale scalability tests, comparing against random splitting, DataSAIL, and Hestia. Results show that ReLaG gives test estimates closer to production performance on synthetic data, controls leakage comparably to existing relation-aware methods on real data, and is much more efficient at large scale. The threshold selection and effective dataset size metrics are also preliminarily validated.
Critical strengths
-
Scalability makes relation-aware splitting practical for large-scale data. ReLaG builds an approximate proximity graph with HNSW, avoiding the quadratic cost of pairwise comparison, and then splits at the community level as atomic units. Existing relation-aware methods are computationally infeasible on large biochemical datasets, and this advantage directly determines the paper's practical value and supports acceptance.
-
Practical label-free threshold selection and OOD detection. The paper uses unlabeled production data and the KS statistic to select the splitting threshold, and produces an OOD signal when alignment fails. Production labels are usually unavailable in real settings, and threshold selection is a key obstacle for deploying relation-aware splitting. This mechanism makes the method closer to realistic evaluation workflows.
Critical weaknesses
-
Limited validation of the approximation-induced leakage claim. ReLaG relies on an HNSW approximate proximity graph, but the paper does not compare its edge recall or inferred communities against an exact graph. It is therefore unclear how missing edges affect community boundaries and residual cross-split leakage. The synthetic experiments show strong group recovery, but they rely on an idealized setting generated by the method's own assumptions. Without a direct comparison, the central claim that communities as atomic units prevent cross-split dependence remains unverified, leaving the reliability of the core technical contribution uncertain.
-
Insufficient validation of deployment-aware threshold selection. The KS-based threshold selection is a main contribution, but its connection to actual deployment performance is not fully established. Real benchmarks lack labeled production data, and the threshold selection is mainly validated on synthetic data generated under the method's own assumptions. It therefore remains unclear whether the KS-selected threshold consistently provides the most accurate test-to-production performance estimates, which leaves the practical effectiveness of the threshold-selection procedure insufficiently established.
Questions for authors
-
Can you compare the HNSW approximate graph with an exact proximity graph on smaller datasets, reporting edge recall, community agreement, and residual cross-split leakage? This would clarify how graph approximation affects splitting reliability.
-
On the synthetic benchmark with labeled production data, how does the KS-selected threshold compare with an oracle threshold that minimizes the test-production performance gap? This would provide more direct evidence for the threshold-selection criterion.
-
How sensitive are the inferred communities and leakage rates to the choice of distance function and null model? In particular, does using the modularity fallback instead of the estimated CPM null model substantially change the splitting results?
2 comments
Thank you for the response and additional experiments, which address most of my concerns. However, the KS-based threshold selection remains insufficiently validated in real-world deployment settings, as the evidence is still primarily based on synthetic data. I will maintain my rating of Weak Acceptance.
Clarification and additional experiments
We appreciate the reviewer’s focus on validating ReLaG’s core components, which led us to three new analyses: a direct evaluation of the HNSW approximation (Appendix 4.1), a threshold sweep on synthetic data comparing the selected threshold with the test–production performance gap (Appendix 3.2), and a sensitivity analysis of the community-detection objective and null model (Appendix A.5). We also clarified the hypothesis underlying threshold selection (Section 4.2). We address each point below.
We thank the reviewer for this suggestion. We added a direct comparison between the HNSW and exact proximity graphs on real molecular and peptide datasets (Appendix 4.1, Table 3, Figure 14). The approximation therefore has a negligible effect on split quality, and treating communities as indivisible units prevents cross-split dependence as effectively as with the exact graph.
Indeed, we show that ReLaG works under the provided data-distribution assumptions. Unlike previous relation-aware methods, ReLaG makes its assumptions explicit, namely that the data arise from latent groups of related samples (Section 3). These assumptions circumscribe the setting in which the method is expected to work.
We thank the reviewer for this suggestion and have added this evaluation (Appendix 4.1). The HNSW graph closely matches the exact proximity graph, and the approximation has a negligible effect on the resulting splits.
We thank the reviewer for this suggestion, which we have addressed in a new experiment (Appendix 3.2). Sweeping the threshold on the synthetic benchmark shows that the test–production gap is close to zero over a wide range of thresholds rather than at a single oracle value. The KS-selected threshold falls within this range, as does the entire plateau minimizing the KS statistic.
We added a sensitivity analysis (Appendix A.5). The sensitivity of communities and leakage rate depends on the relational structure of the dataset. Modularity is a useful fallback, as it substantially reduces leakage relative to random splitting, but it remains less effective than CPM. The distance function, by contrast, defines which samples are considered related and follows from domain knowledge, as discussed in our response to the third weakness.
We hope these additions address the reviewer’s concerns and would be happy to clarify any remaining points.