acceptodds
Under review as a conference paper at ICLR 2027

ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations

Abstract

Random splitting can yield non-independent train-test subsets when a dataset contains related samples, as is common in certain applications such as biochemical studies. This leads to overly optimistic generalization estimates. Here, we introduce ReLaG, a modality-agnostic framework that models sample relatedness through a hierarchical latent-variable process and infers groups of related samples using proximity graphs and community detection to produce independent train-test subsets. Across molecular and protein datasets, ReLaG matches existing relation-aware methods while scaling substantially better, enabling splits at previously impractical dataset sizes. We further introduce a label-free procedure that adapts the splitting resolution to production data, aligning evaluation with the intended deployment setting. ReLaG's inferred groups provide a cheap estimate of effective dataset size, enabling diversity-aware dataset scaling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
ai for science, benchmark, drug discovery, datasets and benchmarks, biology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7oVEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReLaG, a scalable and modality-agnostic relation-aware data splitting framework. It addresses data leakage and overestimated generalization caused by related samples crossing train-test boundaries under random splitting. The method models sample dependence with a two-level latent-variable model, builds an approximate proximity graph with HNSW, infers related sample groups via Leiden community detection, and splits data at the community level as atomic units. It also introduces an adaptive threshold selection method based on unlabeled production data and the KS statistic, which can detect out-of-distribution data and estimate an effective dataset size. Experiments cover synthetic data, real molecular and protein datasets, and large-scale scalability tests, comparing against random splitting, DataSAIL, and Hestia. Results show that ReLaG gives test estimates closer to production performance on synthetic data, controls leakage comparably to existing relation-aware methods on real data, and is much more efficient at large scale. The threshold selection and effective dataset size metrics are also preliminarily validated.

Critical strengths
  1. Scalability makes relation-aware splitting practical for large-scale data. ReLaG builds an approximate proximity graph with HNSW, avoiding the quadratic cost of pairwise comparison, and then splits at the community level as atomic units. Existing relation-aware methods are computationally infeasible on large biochemical datasets, and this advantage directly determines the paper's practical value and supports acceptance.

  2. Practical label-free threshold selection and OOD detection. The paper uses unlabeled production data and the KS statistic to select the splitting threshold, and produces an OOD signal when alignment fails. Production labels are usually unavailable in real settings, and threshold selection is a key obstacle for deploying relation-aware splitting. This mechanism makes the method closer to realistic evaluation workflows.

Critical weaknesses
  1. Limited validation of the approximation-induced leakage claim. ReLaG relies on an HNSW approximate proximity graph, but the paper does not compare its edge recall or inferred communities against an exact graph. It is therefore unclear how missing edges affect community boundaries and residual cross-split leakage. The synthetic experiments show strong group recovery, but they rely on an idealized setting generated by the method's own assumptions. Without a direct comparison, the central claim that communities as atomic units prevent cross-split dependence remains unverified, leaving the reliability of the core technical contribution uncertain.

  2. Insufficient validation of deployment-aware threshold selection. The KS-based threshold selection is a main contribution, but its connection to actual deployment performance is not fully established. Real benchmarks lack labeled production data, and the threshold selection is mainly validated on synthetic data generated under the method's own assumptions. It therefore remains unclear whether the KS-selected threshold consistently provides the most accurate test-to-production performance estimates, which leaves the practical effectiveness of the threshold-selection procedure insufficiently established.

Questions for authors
  1. Can you compare the HNSW approximate graph with an exact proximity graph on smaller datasets, reporting edge recall, community agreement, and residual cross-split leakage? This would clarify how graph approximation affects splitting reliability.

  2. On the synthetic benchmark with labeled production data, how does the KS-selected threshold compare with an oracle threshold that minimizes the test-production performance gap? This would provide more direct evidence for the threshold-selection criterion.

  3. How sensitive are the inferred communities and leakage rates to the choice of distance function and null model? In particular, does using the modularity fallback instead of the estimated CPM null model substantially change the splitting results?

2 comments
AuthorsOpenReview ↗

Clarification and additional experiments

We appreciate the reviewer’s focus on validating ReLaG’s core components, which led us to three new analyses: a direct evaluation of the HNSW approximation (Appendix 4.1), a threshold sweep on synthetic data comparing the selected threshold with the test–production performance gap (Appendix 3.2), and a sensitivity analysis of the community-detection objective and null model (Appendix A.5). We also clarified the hypothesis underlying threshold selection (Section 4.2). We address each point below.

  1. Limited validation of the approximation-induced leakage claim. ReLaG relies on an HNSW approximate proximity graph, but the paper does not compare its edge recall or inferred communities against an exact graph. It is therefore unclear how missing edges affect community boundaries and residual cross-split leakage. The synthetic experiments show strong group recovery, but they rely on an idealized setting generated by the method's own assumptions. Without a direct comparison, the central claim that communities as atomic units prevent cross-split dependence remains unverified, leaving the reliability of the core technical contribution uncertain.

We thank the reviewer for this suggestion. We added a direct comparison between the HNSW and exact proximity graphs on real molecular and peptide datasets (Appendix 4.1, Table 3, Figure 14). The approximation therefore has a negligible effect on split quality, and treating communities as indivisible units prevents cross-split dependence as effectively as with the exact graph.

  1. Insufficient validation of deployment-aware threshold selection. The KS-based threshold selection is a main contribution, but its connection to actual deployment performance is not fully established. Real benchmarks lack labeled production data, and the threshold selection is mainly validated on synthetic data generated under the method's own assumptions. It therefore remains unclear whether the KS-selected threshold consistently provides the most accurate test-to-production performance estimates, which leaves the practical effectiveness of the threshold-selection procedure insufficiently established.

Indeed, we show that ReLaG works under the provided data-distribution assumptions. Unlike previous relation-aware methods, ReLaG makes its assumptions explicit, namely that the data arise from latent groups of related samples (Section 3). These assumptions circumscribe the setting in which the method is expected to work.

Can you compare the HNSW approximate graph with an exact proximity graph on smaller datasets, reporting edge recall, community agreement, and residual cross-split leakage? This would clarify how graph approximation affects splitting reliability.

We thank the reviewer for this suggestion and have added this evaluation (Appendix 4.1). The HNSW graph closely matches the exact proximity graph, and the approximation has a negligible effect on the resulting splits.

On the synthetic benchmark with labeled production data, how does the KS-selected threshold compare with an oracle threshold that minimizes the test-production performance gap? This would provide more direct evidence for the threshold-selection criterion.

We thank the reviewer for this suggestion, which we have addressed in a new experiment (Appendix 3.2). Sweeping the threshold on the synthetic benchmark shows that the test–production gap is close to zero over a wide range of thresholds rather than at a single oracle value. The KS-selected threshold falls within this range, as does the entire plateau minimizing the KS statistic.

How sensitive are the inferred communities and leakage rates to the choice of distance function and null model? In particular, does using the modularity fallback instead of the estimated CPM null model substantially change the splitting results?

We added a sensitivity analysis (Appendix A.5). The sensitivity of communities and leakage rate depends on the relational structure of the dataset. Modularity is a useful fallback, as it substantially reduces leakage relative to random splitting, but it remains less effective than CPM. The distance function, by contrast, defines which samples are considered related and follows from domain knowledge, as discussed in our response to the third weakness.

We hope these additions address the reviewer’s concerns and would be happy to clarify any remaining points.

Thank you for the response and additional experiments, which address most of my concerns. However, the KS-based threshold selection remains insufficiently validated in real-world deployment settings, as the evidence is still primarily based on synthetic data. I will maintain my rating of Weak Acceptance.

Reviewer_xCSdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a train-test splitting method that models sample dependencies through a hierarchical latent-variable model and constructs dataset splits using approximate nearest-neighbor graphs and community detection to reduce dependencies between training and test samples. Experiments on molecular and protein datasets show that the proposed method achieves comparable evaluation quality to existing approaches while offering substantially better scalability.

Critical strengths

The paper focuses on a long-standing issue in machine learning evaluation: dependencies between training and test samples. Such dependencies can lead to overly optimistic estimates of generalization performance when random splitting is used. The main strengths of this work are its reasonable theoretical formulation and clear presentation of the method. The experiments cover multiple datasets and demonstrate substantial improvements in computational efficiency over existing relation-aware splitting methods.

Critical weaknesses

My concerns about this paper mainly focus on three aspects.

First, the methodological novelty is limited. Relation-aware data splitting has been extensively studied, and the proposed method mainly combines established techniques, including approximate nearest-neighbor search, graph-based community detection, and group-level splitting. Compared with existing approaches such as DataSAIL and Hestia, the work does not appear to introduce a substantial technical breakthrough.

Second, I am concerned that excessively separating dependencies and correlations between training and test samples may introduce substantial distribution shifts. Although the synthetic experiments show that ReLaG produces test performance close to that observed in the production setting, real-world datasets often lack labeled production data. Therefore, it remains unclear whether the resulting distribution shifts genuinely reflect deployment conditions or are largely artifacts introduced by the splitting procedure itself.

Third, the theoretical formulation assumes that the inferred communities correspond to independent underlying latent groups. However, in practice, these communities are estimated using distance thresholds and graph clustering, which may not accurately recover the true dependency structure. Different feature representations or distance metrics could lead to substantially different estimates of sample dependencies. The impact of such grouping inaccuracies on the validity of the resulting evaluation remains insufficiently investigated.

Questions for authors

Beyond computational efficiency, what fundamental methodological advantages does ReLaG offer over existing relation-aware splitting methods such as DataSAIL and Hestia?

What evidence supports that the distribution shifts introduced by ReLaG reflect real-world deployment conditions rather than artifacts of the splitting procedure, particularly on real-world datasets?

How robust are the inferred communities and evaluation results to different feature representations, distance metrics, and errors in community detection?

5 comments
AuthorsOpenReview ↗

Clarification and additional experiments [Part 1]

We thank the reviewer for their review. In response, we added two experiments: a distribution-shift test on the eight real-world datasets (Appendix 7.2, Fig. 17b), and a robustness analysis under simulated community-detection errors (Appendix 4.2, Figure 15). We also revised the abstract, introduction, contributions, and conclusion to clarify ReLaG’s methodological contributions beyond scalability. We address each point below.

First, the methodological novelty is limited. Relation-aware data splitting has been extensively studied, and the proposed method mainly combines established techniques, including approximate nearest-neighbor search, graph-based community detection, and group-level splitting. Compared with existing approaches such as DataSAIL and Hestia, the work does not appear to introduce a substantial technical breakthrough.

ReLaG builds on established components, but repurposes the HNSW construction algorithm to build an approximate proximity graph rather than for nearest-neighbor search (Section 4.1). Its main contributions are a latent-variable model that, to our knowledge, provides the first theoretical framework supporting graph-based splitting (Section 3), and the first data-driven threshold-selection procedure, which also recovers random splitting and flags out-of-distribution data (Sections 4.2 and 5.3). Although ReLaG shares with DataSAIL and Hestia the goal of preventing leakage, which motivates their comparison, the three methods pursue different objectives (Section 2, last paragraph). We agree with the reviewer that the original manuscript may have emphasized scalability and overshadowed these contributions. We have therefore revised the abstract, introduction, and conclusion to make the contribution clearer. For instance, the introduction’s contributions list now include “We introduce the first data-driven threshold-selection procedure, which adapts the splitting resolution to the intended production data without requiring labels,” and the conclusion adds “The latent-variable model thus provides a principled basis for graph-based splitting, justifying its effectiveness and making its scope explicit through the model’s assumptions.”

Second, I am concerned that excessively separating dependencies and correlations between training and test samples may introduce substantial distribution shifts. Although the synthetic experiments show that ReLaG produces test performance close to that observed in the production setting, real-world datasets often lack labeled production data. Therefore, it remains unclear whether the resulting distribution shifts genuinely reflect deployment conditions or are largely artifacts introduced by the splitting procedure itself.

This is a valuable point, which we tested in a new experiment (Appendix 7.2, Figure 17b): a model cannot distinguish between the two subsets of a ReLaG split, performing at chance level like a random-split control. This suggests that ReLaG does not introduce a detectable distribution shift on real data, and that its lower test scores reflect reduced leakage rather than an artifact of the splitting procedure.

Third, the theoretical formulation assumes that the inferred communities correspond to independent underlying latent groups. However, in practice, these communities are estimated using distance thresholds and graph clustering, which may not accurately recover the true dependency structure. Different feature representations or distance metrics could lead to substantially different estimates of sample dependencies. The impact of such grouping inaccuracies on the validity of the resulting evaluation remains insufficiently investigated.

In practice, the representation and distance function are not choosen arbitrarly: they follow from domain knowledge about how related samples arise, with well-established choices such as sequence alignment for proteins, Tanimoto similarity for molecules or TM for 3d structures. It is true, however, that whatever the distance function and representation, the inferred communities remain estimates of the latent groups. We therefore added an experiment simulating community-detection errors (Appendix 4.2, Figure 15), which shows a resilience phase as the evaluation remains essentially unchanged up to about a tenth of the communities altered, and then degrades gradually towards random splitting.

AuthorsOpenReview ↗

Clarification and additional experiments [Part 2]

Beyond computational efficiency, what fundamental methodological advantages does ReLaG offer over existing relation-aware splitting methods such as DataSAIL and Hestia?

Please see our response to the first weakness and the contributions listed in Section 1. In brief, beyond scalability, ReLaG provides a theoretical framework grounding graph-based splitting (Section 3) and the first data-driven threshold-selection procedure, which also recovers random splitting and flags out-of-distribution data (Sections 4.2 and 5.3).

What evidence supports that the distribution shifts introduced by ReLaG reflect real-world deployment conditions rather than artifacts of the splitting procedure, particularly on real-world datasets?

We found no evidence that ReLaG introduces a distribution shift (Response to weakness 2, Appendix 5.2, Figure 14b). This is expected under the latent-variable model, since groups are drawn independently from the same population-level distribution, so randomly assigning entire groups to train or test yields subsets from the same distribution (Section 3). Moreover, the threshold-selection procedure explicitly aims to match the distance distributions of training and production data, so that test conditions reflect deployment conditions rather than being set arbitrarily (Section 4.2).

How robust are the inferred communities and evaluation results to different feature representations, distance metrics, and errors in community detection?

Please see our response to the third weakness. In brief, the representation and distance function follow from domain knowledge rather than acting as free hyperparameters, and the evaluation remains essentially unchanged up to about a tenth of the communities altered by simulated detection errors (Appendix 4.3, Figure 13).

We hope these clarifications and additions address the reviewer’s concerns, and we would be happy to answer any further questions.

Thank you for the rebuttal and additional experiments. The distribution-shift and community-corruption analyses are useful, but two key concerns remain.

Q1. Contribution of threshold selection. The paper proposes KS-based threshold selection as a main contribution, but Appendix A.3.2 shows that the test–production performance gap remains close to zero across a wide threshold range (0.15–0.70). What additional benefit does selecting the KS-minimizing threshold provide over fixed thresholds or simpler data-driven alternatives? Can the authors quantify this benefit in terms of test–production performance gap or leakage?

Q2. Robustness to distance metrics. Figure 15 evaluates random community splitting and merging, but does not address errors caused by the distance function itself. Since ReLaG relies on distance-based graph construction, could the authors compare two reasonable distance metrics or representations on the same dataset and report their effects on community assignments, train–test leakage, and downstream performance?

These questions concern the independent contribution and reliability of the proposed method. While the rebuttal improves the paper, I maintain my rating of weak rejection and welcome further clarification.

Reviewer_dYS5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper considers proper design of training and testing data in the biological domain, as having closely related train-test sample pairs may give overly optimistic performance estimate that does not reflect generalization to real world applications.

The proposed approach, Relation-base Latent Grouping (ReLaG), is a modality independent method for data partitioning. The method assumes that samples may be related and organized into groups, such as protein super-families or drug hits in biology. Each group may be represented with an unobserved group-specific latent variable, while samples are drawn from the group and conditioned the group's latent representation. A good dataset split is assumed to work on group boundaries, i.e. an entire group should either be used for train, validation or test.

ReLaG assumes an appropriate embedding space and distance function and uses them to build a proximity graph, with samples representing the nodes connected by edges only if the distance is less than a given threshold. To improve efficiency, only an approximate proximity graph is built using Hierarchical Navigable Small World data structure that organizes samples into increasingly sparser graph layers. After the graph is built it may be used to detect groups.

Critical strengths

[S1] Improving experimental design is an important topic, especially in the age of foundation models, where it is not clear how much the test data is leaking into the training, or how exactly to test for robustness and generalization.

[S2] The proposed approach is more efficient compared to alternatives.

Critical weaknesses

[W1] The method depends on a good quality embedding space and distance metric. It is not clear if those exist for any problem or how exactly they are found. This potentially limits the reach of the paper

[W2] The method is significantly impacted by the choice of distance threshold for determining proximity, and will impact group coarseness. The proposed method of choosing the threshold assumes the availability of production data which seems overly optimistic and does not correspond to most of the experiments done on real data..

[W3] The results suggest that randomly choosing training data gives a big discrepancy between test and production results, it is not necessarily a problem if validation/test results are helpful in choosing the best performing model (i.e. the ranking of performance is what is important). Also, it is not clear that ReLaG is significantly better than other non-random methods for splitting data.

Questions for authors
  1. It would be helpful to indicate what is the expected indication of ReLaG advantage over competing methods (PCC better when lower or higher?)
  2. Why is MNIST a suitable to show threshold choice for i.i.d examples? The sampling process was impacted by the choice of classes, so the model would not work on e.g. dog or cat images? That is somewhat analogous to the biological setup where families might appear in production that were not considered during training.
  3. Could you show if ReLaG has better model ranking than training on random data, that is to say does random training also choose a suboptimal model because test performance is optimistic relative to production performance?
3 comments
AuthorsOpenReview ↗

Clarification and additional experiments [Part 1]

We thank the reviewer for their assessment and constructive questions. In response, we added an experiment on model ranking (Appendix A.7.1, Fig. 17a), clarified the interpretation of PCC in the caption of Fig. 6, and revised the limitations (Sec. 6). We address each point below.

[W1] The method depends on a good quality embedding space and distance metric. It is not clear if those exist for any problem or how exactly they are found. This potentially limits the reach of the paper.

It is true that relying on domain-specific distance metrics and embeddings is a limitation in some contexts. This is, however, standard in the field and also what is used by the strong baselines DataSAIL (Joeres et al., 2025) and Hestia (Fernandez-Diaz et al., 2024a). We revised the conclusion to clarify this limitation: “Similar to other relation-aware methods, ReLaG requires an explicit distance function informative of relatedness, which is readily available for many, but not all, modalities.” Note that this design choice allows domain experts to control data splitting in accordance with the task-specificities, which enforces confidence in the model evaluation on the provided test set. Importantly, biologists and chemists commonly rely on domain-specific distances, such as sequence identity or Tanimoto similarity, both used in our experiments on real data.

[W2] The method is significantly impacted by the choice of distance threshold for determining proximity, and will impact group coarseness. The proposed method of choosing the threshold assumes the availability of production data which seems overly optimistic and does not correspond to most of the experiments done on real data.

Unlabeled production data are often much cheaper to obtain than labels and are commonly available when a model is intended for a specific application, for instance the catalog of candidate molecules a model will be used to prioritize for laboratory testing in drug discovery (Section 4.2). They are lacking mainly when models are trained without an immediate deployment target. When unavailable, the threshold can be set manually, as with other relation-aware methods. This has been clarified in the limitation section: “Automatic threshold selection further requires unlabeled production data, which are not always available. In their absence, the threshold can be set manually, as with other relation-aware methods.”

[W3] The results suggest that randomly choosing training data gives a big discrepancy between test and production results, it is not necessarily a problem if validation/test results are helpful in choosing the best performing model (i.e. the ranking of performance is what is important). Also, it is not clear that ReLaG is significantly better than other non-random methods for splitting data.

We agree that the test set is often used to rank and select models. However, leakage can distort the ranking itself. Random splits reward memorization of near-duplicates, so they can favor models or checkpoints that exploit similarity to the training set over those that generalize. To test this, we added an experiment (Appendix 7.1, Figure 17a) on the synthetic dataset. We evaluate four checkpoints of the same training run, from under-trained to over-fitted to represent different model capacities. In production, performance peaks at 10 epochs and slightly declines afterward. The random test set misses this decline: its PCC keeps increasing up to 100 epochs making it select the overfitted checkpoint. The ReLaG test set follows the production curve and selects the same checkpoint as production. This result suggests that relation-aware splits can help select better models.

Regarding the second point, we do not claim that ReLaG is significantly better than other relation-aware methods. We claim comparable split quality (Lines: 62, 329, 475, initial manuscript) at substantially lower computational cost (Figure 7: 1.7 h vs. an extrapolated 2.1 weeks for Hestia at 3.1M samples), which enables relation-aware splitting at scales where existing methods are impractical.

AuthorsOpenReview ↗

Clarification and additional experiments [Part 2]

It would be helpful to indicate what is the expected indication of ReLaG advantage over competing methods (PCC better when lower or higher?)

A higher test Pearson correlation (PCC) usually indicates a better model, but under train–test leakage it can reflect memorization rather than generalization. A faithful evaluation is therefore one whose test PCC reflects production performance (Figure 5b); we have clarified this in the caption of Figure 6 by adding this sentence: “Lower scores than random splitting are consistent with the removal of train–test leakage.”

Why is MNIST a suitable to show threshold choice for i.i.d examples? The sampling process was impacted by the choice of classes, so the model would not work on e.g. dog or cat images? That is somewhat analogous to the biological setup where families might appear in production that were not considered during training.

In MNIST, the classes are the prediction target and are fixed: all ten digits appear in training and in production, so seeing a “7” during training is not leakage. Biochemical families as latent variables have diffuse boundaries, and the total number of families is unknown. Many of them are unobserved during training. The task is to predict a biochemical property, not the family, which requires generalizing to unseen families.

Could you show if ReLaG has better model ranking than training on random data, that is to say does random training also choose a suboptimal model because test performance is optimistic relative to production performance?

Yes. As mentioned to our answer to [W3] above, we added this experiment (Appendix 7.1, Figure 17a): the random test set selects an overfitted checkpoint, whereas the ReLaG test set selects the checkpoint that performs best in production

We hope these clarifications and additions address the reviewer’s concerns, and we would be happy to answer any further questions.

We thank the reviewer for their thoughtful feedback. As the discussion period nears its end, we would be grateful to hear whether our responses have addressed the reviewer’s main concerns, or whether there are any remaining points we could clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Improvements made to the uploaded manuscript following reviewer comments

We thank the reviewers for their questions and comments, which helped us improve the manuscript. For reference, we list below the sections that were revised or added in the updated version.

Clarifications:

  • Abstract
  • Introduction (lines 66-71; 81)
  • Proposed methods (line 195; lines 252-261)
  • Caption of Figure 6
  • Evaluation (line 403)
  • Conclusion (lines 480-493)

New experiments:

  • Oracle validation of selected threshold (A.3.2), following Reviewer 7oVE question
  • Design choices evaluation (A.4), following Reviewer 7oVE and xCSd comments
  • Leiden objective and resolution sensitivity (A.5), following Reviewer 7oVE question
  • Additional test-production comparisons (A.7), following Reviewer dYS5 and xCSd comments

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.