acceptodds
Under review as a conference paper at ICLR 2027

Tuned Convex Augmentation Beats the Sample It Is Built From

Abstract

Data-augmentation methods such as SMOTE and mixup create synthetic points as random convex combinations of real examples. We ask when the resulting synthetic distribution is closer to the data distribution, in Wasserstein distance, than the real examples it is built from, whose empirical distribution converges at rate . Mixing nearby points smooths the sample. This reduces fluctuation at the cost of a bias of order , where is the neighbourhood size. For a smooth density bounded away from zero, we prove that -NN SMOTE with tuned beats for every in every dimension . Its rate reaches once , and for a non-flat density on the torus no local mixing scheme with a bias of order can do better. The gain carries over to a finite training set only if synthetic points make up most of it. In experiments, tuned mixing of pairs beats the real sample for as small as . In full dimension a tuned kernel does better, while on the two manifolds we test mixing beats an isotropic kernel at every sample size and ambient dimension.

Primary area:
learning theory
Keywords:
Mixup, SMOTE, convex augmentation, Wasserstein distance, effective sample size

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5DF5OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1(−1)
Read review
Paper summary

The paper studies whether convex data augmentation methods, such as SMOTE, can generate synthetic distributions that approximate the true data distribution better than the original empirical samples. The authors provide theoretical convergence rates in Wasserstein distance, analyze the effect of neighborhood size and mixing bias, and compare the proposed approach with kernel smoothing using synthetic experiments.

Critical strengths

The paper addresses an interesting theoretical question about data augmentation. The authors provide upper and lower bounds for the convergence rates and analyze the trade-off between smoothing and bias. I also appreciate that the paper discusses some limitations of the theoretical results, particularly when using a finite number of synthetic samples.

Critical weaknesses

My main concern is the overall clarity and organization of the paper. I found the paper extremely difficult to read and follow, especially given the large number of theorems, propositions, and technical arguments. Although these results are mathematically related, I feel that the authors do not sufficiently explain how each result contributes to the main research question. As a result, the central contribution is difficult to identify, and the paper becomes unnecessarily complicated.

I also think the paper could be much better structured. The authors introduce several theoretical directions, including convergence rates, mixing bias, kernel smoothing, and manifold analysis, but the motivation for moving between these topics is not always clear. I believe the paper would benefit from a more focused presentation that emphasizes the most important results and explains their significance before introducing additional technical details.

Another concern is the experimental evaluation. The experiments are limited to synthetic distributions, and I am not convinced that the results demonstrate the practical usefulness of the analysis. In particular, it is unclear whether the improvements in Wasserstein distance translate into better performance on real-world machine learning tasks.

Moreover, some experiments use oracle tuning based on the true distribution, which is generally unavailable in practice. Although the authors also evaluate a theoretical tuning rule, I think a fully data-driven evaluation would provide stronger evidence.

I am also concerned about the gap between the theoretical setup and the experiments. The main convergence theorem assumes sample splitting, while the experiments use unsplit samples. In addition, the manifold experiments are not supported by an equivalent general convergence-rate theorem.

Finally, I could not find a public code repository in the provided manuscript, which makes independent reproduction of the experimental results more difficult, despite the detailed experimental descriptions in the appendix.

Questions for authors
  • Could the authors explain more clearly what the main theoretical contribution is and why all the additional theoretical results are necessary?

  • Could the authors provide a clearer connection between the different theorems and explain how each one supports the central claim?

  • Why are the experiments limited to synthetic distributions? Could the authors provide results on real-world datasets or downstream learning tasks?

  • How can the neighborhood size be tuned in practice without access to the true distribution?

  • Could the authors clarify how the theoretical guarantees apply to the unsplit sampling procedure used in the experiments?

  • Given Corollary 3.1, which shows that the improved convergence rate may not carry over to finite augmented training sets, how significant is the main result for practical data augmentation?

Reviewer_HTaqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies convex data-augmentation methods such as SMOTE and mixup, which create synthetic points as random convex combinations of nearby real examples. It asks when the resulting synthetic distribution is closer to the data distribution, in Wasserstein distance, than the empirical distribution of the real sample. For a smooth density, the authors prove that k-NN SMOTE with a tuned k beats the empirical rate n^(-1/d) in every dimension d ≥ 3. They also give a matching lower bound on the torus for local convex mixing schemes with bias of order h². The analysis is complemented by a computation of the mixing bias, an analysis of the conditions under which a finite synthetic training set retains the gain, a comparison with kernel smoothing on manifolds, and synthetic experiments.

Critical strengths

The results seem sound, and the positioning relative to other related work provides a complete picture.

Critical weaknesses

The paper is very hard to read and to understand from the start. There is partial background on the problem: why it is important, what the challenges are, and why they arise. For example, which downstream tasks might benefit from this. The main paper is overloaded with mathematical notation and explicit parameter values, which makes it nearly unreadable. In my opinion, it requires a deep edit.

Questions for authors

Can the authors explain the motivation more clearly: why is it important to know when the synthetic distribution is closer to the data distribution than the real sample, and what are the main challenges in showing this?

Reviewer_EPJFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper derives a bound for the Wasserstein distance between synthetic distribution generated by convex augmentation methods and the target distribution. It proves that this synthetic distribution converges faster than the empirical measure. Synthetic experiments on toy data, in several settings, are done to compare convergence rates of Mixup vs other augmentation techniques, such as kernel smoothing.

Critical strengths
  • The paper provides a strong theoretical explanation of convex augmentation techniques.

  • The paper provides upper and lower bounds for the Wasserstein distance between synthetic distribution (generated by convex augmentation method such as SMOTE) and target distribution.

  • Theoretical results comparing kernel augmentation and convex augmentation on distributions on manifold.

  • The paper provide some guidelines for hyper-parameter selection in SMOTE.

  • Experiments on several toy settings, e.g. manifold data, comparing convex augmentation techniques with kernel smoothing.

Critical weaknesses
  • The setting is not the same than mixup done in practice, mainly because of the split of the dataset in two distinct parts. What are the implications of this difference?

  • The experiments are on toy datasets. Although I understand the main purpose of the paper is the theoretical motivation of convex augmentation methods, the experimental part is a bit weak. Most notably, the authors provide some guidelines to tune hyper-parameters. It would have been interesting to verify that these guidelines are useful in more complex settings.

Questions for authors

see weaknesses

Comments on the paper

2 comments

We thank the reviewer for reading the rebuttal and for raising the score. Each of the three remaining points has a concrete answer in the revision.

1. Downstream performance. We measure W1W_1 because it controls downstream performance for every Lipschitz loss at once. By Corollary 3.2, a model trained on the synthetic distribution has excess risk at most a constant times LW1(ν^,μ0)LW_1(\hat\nu,\mu_0) for any loss that is LL-Lipschitz in the features. So a 4–14% smaller W1W_1 on the real data (Table 26) is a 4–14% smaller bound on the downstream loss for all such losses and models, whereas a classifier run would show one loss and one model. The ordering of KK given by our theory is also the one observed for F1F_1 on 21 imbalanced datasets by Kachan et al. (2025).

2. Theory–practice gap. We believe the revision closes this gap. The cross-fit generator of Section 2, which runs the split generator twice with the halves exchanged and averages, uses every point, has the proved rate, and is as simple to run as SMOTE. It is a practical method with a guarantee, and the split itself is only a device of the proofs. What remains unproved is the rate of standard SMOTE itself, which differs from the proved generator in that the neighbours of a point are searched among the other half of the sample, and which behaves identically in our runs: on the same samples with the same kk, the exponents are within 0.023 and the levels within 11% of each other (Table 12). The manifold experiments, which were not covered by a theorem at submission, are now covered by Theorem H.1.

3. Finite augmented sets. Corollary 3.1 answers this exactly. With NN synthetic points the error cannot fall below (n+N)−1/d(n+N)^{-1/d}, so a small augmented set keeps a better constant but not the rate, and the rate is kept when NN grows faster than nn. Oversampling to class parity, the typical use of SMOTE, gives N=(ρ−1)nN=(\rho-1)n for an imbalance ratio ρ\rho, which is the case where the condition can hold. The real-data experiment is a finite augmented set of this kind, with ρ\rho from 9 to 42 and a 4–14% gain. The rule for kk and the threshold kdk_d hold at any NN.

Presentation. We take the point, and we have uploaded a restructured version rather than promising one. The main change is that the sharpest result is now stated in the main text, as part (iii) of Theorem A: with its own kk, kk-NN SMOTE attains the exponent eK=min⁡[2/(d+2),2K/(d(K+1))]e_K=\min[2/(d+2),2K/(d(K+1))] at every KK, so that with Theorem B the exponent of local convex mixing is known exactly. Section 3 now contains Theorems A and B, their two corollaries and one paragraph of mechanism in words, and the technical discussion (the chaining argument below the sampling scale and the proof ideas) has moved to the new Appendix E.7. The manifold extension is stated in the main text as Theorem C. The abstract and the title of Theorem A now name the split form of the generator, and Section 6 says that the real-data gains are gains in the constant, outside the regime of the rate theorem. The results themselves do not change.

If any of the three points is still open after this reply, we would be glad to address it before the discussion closes.

AuthorsOpenReview ↗

Summary of the discussion

We thank the three reviewers for their time and for the questions, which have improved the paper. Below we summarize what the paper proves, why we believe it is new, and how each concern raised during the review was addressed.

What the paper proves and why it is new. SMOTE-type augmentation builds synthetic points as convex combinations of KK nearby real points. The paper gives the first convergence rate for the resulting synthetic distribution to the data distribution, in W1W_1, together with a matching lower bound. Theorem A shows that split kk-NN SMOTE with kk from an explicit rule converges faster than the real sample it is built from, for every K≥2K\ge2 and d≥3d\ge3, and that with the optimal kk it attains the exponent eK=min⁡[2/(d+2),2K/(d(K+1))]e_K=\min[2/(d+2),2K/(d(K+1))]. Theorem B shows that no local convex mixing scheme with the usual bias of order h2h^2 does better. The exponent of local convex mixing is therefore known exactly. The mechanism is new as well. Mixing smooths only along the K−1K-1 directions of each simplex, and the gain depends on how K−1K-1 compares with d−2d-2, a threshold that kernel estimators do not have. The results have direct practical content: a rule for kk that needs no tuning, a threshold kdk_d below which the default k=5k=5 stops helping from about six dimensions, and the ordering of KK. Theorem C extends both theorems to data on a manifold with the intrinsic dimension in place of dd, and Corollaries 3.1 and 3.2 say what the rate means for a finite augmented training set and for the risk of a model trained on it. Earlier theory of SMOTE gives the distribution of one synthetic point, consistency, or risk bounds, but no rate for the synthetic distribution and no lower bound. All three reviewers found the results sound, and no error was reported.

Concerns raised and how they were resolved.

  1. Presentation (Reviewers HTaq and 5DF5). The introduction now states the motivation, the three difficulties of the problem and a map of the paper. The optimal-exponent result is stated in the main text as part 3 of Theorem A, and the manifold extension as Theorem C. The technical discussion of the proofs moved from Section 3 to Appendix E.7, numerical details of Section 6 moved to the appendix, paper-specific terms were replaced by plain ones.

  2. Sample splitting versus practice (Reviewers EPJF and 5DF5). The split is a device of the proofs. The cross-fitted generator of Section 2 uses every point and has the proved rate. On the same samples with the same kk, the split and unsplit generators have exponents within 0.023 and levels within 11% of each other (new Table 12).

  3. Synthetic data only, practical validation (Reviewers EPJF and 5DF5). New Appendix I.3 tests the recommendations on eight real imbalanced datasets of dimension 6 to 100. With the rule and no tuning, every KK brings the synthetic minority class 4–14% closer to held-out points than the sample, against 4–7% for SMOTE with k=5k=5, and K=mK=m is best on every dataset. A multiple of the rule chosen on a validation split is as good as the rule itself (Table 26). W1W_1 bounds the downstream loss for every Lipschitz loss and model at once (Corollary 3.2). We do not report classifier accuracy, and Section 7 says so.

  4. Oracle tuning (Reviewer 5DF5). The untuned rule is reported beside the oracle-tuned kk in every experiment and is within 13% of it, and on the real data the choice is made on validation data only.

  5. Manifold experiments without a theorem (Reviewer 5DF5). Theorem C (proved as Theorem H.1) now covers them.

  6. Finite augmented sets (Reviewer 5DF5). Corollary 3.1 gives the exact condition. A small augmented set keeps a better constant and not the rate, and the rate is kept when synthetic points dominate, which is the case of oversampling to class parity. Section 6 states that the real-data gains are gains in the constant.

  7. Code (Reviewer 5DF5). The code and result files are attached as supplementary material.

We hope that all questions raised during the review have been answered, and we thank the reviewers and the Area Chair for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.