acceptodds
Withdrawn from ICLR 2027

Hilbert Split: Geometry-Aware Data Splitting for Generalization Assessment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data splitting is a commonly used technique for assessing how well predictive models generalize. The challenge is to preserve the distribution of the data relevant to prediction while limiting train–test similarities that can make evaluation overly optimistic. We introduce Hilbert Split, which identifies response-relevant predictors through sensitivity analysis. The local regression fits underlying this analysis provide a geometry for separating observations. A Hilbert ordering of the selected predictors and response organizes the data into local strata, within which this geometry guides test-point selection. This construction preserves distributional coverage while reducing train-test similarity. For randomized within-stratum selection, we establish sharper concentration of summary averages than simple random splitting and Wasserstein control of distributional discrepancy, with rates governed by the selected dimension.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
data splitting, generalization, similarity-aware evaluation, sensitivity analysis, Hilbert curve

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dWZ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hilbert Split, a geometry-aware data splitting method designed to improve the reliability of generalization assessment. The method first identifies response-relevant features through sensitivity analysis, then uses Hilbert curve ordering to construct local strata. Within each stratum, test samples are selected by maximizing train–test separation under a locally weighted distance metric. The paper provides theoretical results on concentration bounds and distributional discrepancy for randomized Hilbert stratification. Experiments on synthetic and real regression datasets demonstrate improved distributional coverage, reduced train–test similarity, and better generalization performance in model selection.

Critical strengths

The paper addresses an important problem in generalization assessment by balancing distributional representativeness and train–test separation. The proposed method is supported by theoretical analysis and experiments on synthetic and real datasets, demonstrating its potential to improve the reliability of model evaluation.

Critical weaknesses

Greater train-test separation does not always lead to better generalization and may introduce overly conservative evaluation. Such motivations are not well-grounded.

Selecting one sample from each stratum will not exclude similar samples. On the other hand, the optimal solution global-wise may not be obtained by the greedy method and its efficiency has not been carefully examined.

Questions for authors

Could the authors please share the rationale for maximizing train-test separation for better generalization?

Reviewer_T66dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This article proposes a method to split a dataset into training and test sets that are both representative and well separated. After a screening step selects the relevant predictors, it orders the points along a Hilbert curve, cuts the ordering into strata, and takes from each stratum the point farthest from the training points. The theory covers a randomised variant, and the experiments use small tabular regression problems.

Critical strengths
  • The problem is real: random splits put near-duplicates on both sides and make the test error optimistic.

  • Combining a locality-preserving Hilbert ordering with a separation criterion is natural.

  • The results on randomised stratification are clear and, as far as I checked, correct. The lower bound on the exponent 1+2/d1+2/d is a nice addition.

  • Screening fixes a real weakness of SPlit and Twinning (Fig. 2).

  • The protocols are detailed, and the authors promise reproduction material.

Critical weaknesses
  • Core components are unpublished. PRIME, on which the method depends, and MaxTwin, a main baseline and the model-selection protocol, are cited only as anonymous manuscripts. App. B gives implementation details of PRIME but no rationale and no validation. Since PRIME defines the strata and the geometry, the contribution cannot be assessed. This alone prevents acceptance.

  • The theory does not cover the algorithm. All guarantees assume one test point drawn independently and uniformly per stratum. Algorithm 1 picks it deterministically, with a sequential maximin rule. None of the results applies to the evaluated method. Maximin also favours sparse regions, which should bias the test error upward.

  • The gain requires very large test sets.

  • The evaluation does not show better risk estimates. Fig. 3 counts a larger test–train gap as a success, without comparing to the true risk. MaxTwin also produces large gaps.

  • The model-selection experiment favours the method. Fig. 5 uses data built so that pairs are close only in the active coordinates, which is what the screening targets. It uses a single design with n=200 and tunes 11 hyperparameters on 40 test points.

Questions for authors

Can you address the five points in "weaknesses"?

Reviewer_KQtROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Hilbert-split, a data splitting method for machine learning aimed at selecting test sets which are simultaneously distributionally representative and sufficiently dissimilar to training sets. The proposed methodology consists of three steps. First, response-relevant features are selected using local linear regression and residual risk comparisons. Second, selected predictors and response are ordered using a Hilbert space-filling to stratify the data. Finally, deterministic min-max procedure is employed to select the test points. Paper also compares the proposed methodology with existing methods on synthetic and real regression problems.

Critical strengths

Paper studies an important machine learning problem and develops an interesting combination of ideas from variable selection and local Hilbert geometry.

Critical weaknesses
  1. It seems that a vast majority of theory is irrelevant to the core contribution of paper. Theory in section 2 involves randomization in stratification whereas procedure discussed in section 4.1 is deterministic.
  2. Sampling procedure developed in paper involves multiples steps, whereas none of the experiments isolate one or more steps and evaluate the contribution of each step.
  3. Paper uses response variable to both select predictors and also in stratification. I want to know how it relates to generalization error.
  4. For experiments in section 5.2 a wider training–test MSE gap does not necessarily indicate that Hilbert Split produces a better test set or provides a more reliable assessment of generalization performance. Similarly, the experiment in section 5.3 involves less than 20 points which makes it difficult to assess the generalizability of observed behavior.
  5. Authors should consider writing small paragraph on notation and also sections 2.1 and 4.1 to give clear explanation of stratification procedure.
Questions for authors

In addition to giving satisfactory responses to above mentioned observations, authors should consider following questions

  1. Can the authors establish any distributional discrepancy guarantees for the actual deterministic sequential selection, rather than only for randomized Hilbert stratification?
  2. Authors should also consider comparisons of computational complexities and practical times.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.