acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Data Scaling Through Data Geometry: A Uniformity Perspective

Abstract

Scaling training data does not necessarily scale the amount of useful learning signal available to a model. We study this discrepancy through the geometry of training data and ask why carefully structured subsets can learn more efficiently than substantially larger datasets. Across controlled regression and LLM fine-tuning, subsets with more uniform geometry optimize faster than equal-size random subsets, and in several settings a half-size subset even outperforms the full dataset, revealing that scaling data can add redundancy faster than useful geometric coverage. We provide two complementary theoretical explanations for this phenomenon. From an optimization perspective, closely spaced examples can induce nearly dependent rows in the model Jacobian, leading to poorly conditioned data geometry and slower convergence under gradient descent. From an approximation perspective, data spacing and local simplex shape determine the conditioning of the interpolation geometry, with more uniform data simplices yielding tighter approximation bounds. Both views connect learning efficiency to data uniformity through the minimum pairwise separation and related geometric quantities, with consistent empirical benefits across datasets, model sizes, and optimization settings. Together, these findings provide a uniformity-based explanation for why training on smaller, geometrically structured subsets can be more efficient than training on larger datasets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data Scaling, Data Uniformity, Learning Efficiency, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N5upOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how data geometry affects training efficiency and approximation error. The authors focus on data uniformity and argue that a smaller and well-distributed dataset can perform better than a larger dataset with redundant samples. The paper develop two theoretical results. The first relates the distance between training samples to the smallest eigenvalue of the Jacobian Gram matrix. The second studies approximation error using local simplex geometry. The authors consider a farthest-first data selection method. The numerical results show that uniform subsets can train faster and achieve similar or better performance than larger datasets.

Critical strengths

The paper studies an important problem. Using more training data does not always improve learning efficiency. The authors try to explain this through both optimization and approximation theory; see Subsections 3.1 and 3.2, pp. 5-6. I find this connection interesting. The discussion of the trade-off between data separation and coverage is also useful.

Critical weaknesses

I have two main concerns.

The first concern is the theoretical result. (i) Theorem 2 (p. 5) gives an upper bound on the smallest eigenvalue of the Jacobian Gram matrix. However, this does not show that increasing the minimum distance between samples will improve convergence. The authors also acknowledge this in the paper but did not deal with it. (ii) Theorem 3 (p. 6) shows that the approximation bound depends on the triangulation and an auxiliary function. But these quantities will change across datasets. Thus, it is difficult to use this result to explain why one selected subset should outperform another.

The second concern is the real performance. If we pay attention to Figure 4 panel (b) on page 8, the results show 43.82% for uniform selection, 43.09% for random selection, and 43.63% for the full dataset. I am not convinced that such small difference matters in practice. The authors report improvements in training time, which is quite obvious given different sample sizes. However, I am wondering whether reaching the same training loss is a fair comparison when the datasets contain different samples.

Questions for authors

My questions will be based on my comments for critical weaknesses.

(1) How does Theorem 2 support the claim that more uniform data lead to faster convergence? It would be great if the authors can provide some numerical evidence showing that uniform selection improves the relevant eigenvalues during training.

(2) Since the approximation bound in Theorem 3 depends on quantities that could change with the selected dataset. how can we use this bound to compare different subsets? I am specifically interested in understanding how this result relates to the neural networks trained in the experiments.

(3) I am confused how the authors calculate the MSE, MAE etc. in the paper. I am not sure whether the numerical results are averaged over multiple runs. Also, was the stopping criterion in Subsection 3.3 used to determine the subset sizes in the experiments?

(4) Figure 1 is obviously generated from AI (correct me if I am wrong). Figure 4 (b) and Figure 5 (b), the legend is out of the figure box. There is no journal details for the citation "Krzysztof Kurdyka and Laurentiu Paunescu. Hyperbolic polynomials and multiparameter realanalytic perturbation theory. 2008." I understand that the authors acknowledge the use of AI for language editing. But please recheck the paper and ensure figures/writings are aligned with academic standards.

Reviewer_5nXxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how training-data geometry affects learning efficiency beyond sample count. It connects near-colliding inputs to weak Jacobian-based convergence bounds and local simplex geometry to approximation bounds. These analyses motivate farthest-first subset selection and a stopping criterion balancing sample separation with local coverage. Validation spans California Housing regression and LLaMA/Qwen instruction tuning on WizardLM, Tulu-3, and LESS-selected data. Compared with equal-size random subsets and full candidate pools, geometrically dispersed subsets reach training-loss thresholds faster and, in several settings, achieve comparable or better downstream performance using roughly half the examples. I view its central contribution as an investigation of how redundancy and geometric coverage shape the value of additional data.

Critical strengths
  • The comparisons with equal-size random subsets the strongest aspect of the paper. Improvements are reported across regression and multiple instruction-tuning settings, including further gains on data already selected by LESS. The latter observation is particularly useful: it suggests that geometric selection can complement model-aware importance scores. These results provide a practical motivation for studying geometric redundancy when reducing training-data requirements.
Critical weaknesses
  1. The theoretical results do not establish the proposed explanation for the performance of the trained models. Theorem 2 gives a local upper bound of the form μlow,s,X=λmin⁡(JsJs⊤)≤Ck,ihmin⁡\mu_{\mathrm{low},s,X}=\lambda_{\min}(J_sJ_s^\top)\le C_{k,i}h_{\min}. This explains how near-collisions can weaken a worst-case convergence guarantee. However, increasing hmin⁡h_{\min} does not imply that μlow,s,X\mu_{\mathrm{low},s,X} increases, and a small minimum eigenvalue does not itself establish slow optimization. For squared loss, writing r=fθ(X)−Yr=f_\theta(X)-Y, the actual gradient magnitude satisfies ∥∇θL∥2=1N2r⊤JJ⊤r\|\nabla_\theta L\|^2=\frac{1}{N^2}r^\top JJ^\top r. Progress therefore depends on the residual’s alignment with the eigenspaces, not just the smallest eigenvalue. The paper acknowledges the one-sided nature of its bound, but does not empirically establish that the selected subsets improve the relevant Jacobian geometry. The connection between Word2Vec separation and the geometry governing LLM fine-tuning consequently remains unverified. The approximation analysis has a related limitation. Theorem 3, formalized as Theorem 4, establishes the existence of a network satisfying ∥f−g∥Lp(I∘)≤3Cmax⁡{HT,1}mmin⁡{sT,1}−r∥φ∥Wm,p(I∘)\|f-g\|_{L^p(I^\circ)} \le 3C\max\{H_{\mathcal T},1\}^{m} \min\{s_{\mathcal T},1\}^{-r} \|\varphi\|_{W^{m,p}(I^\circ)}. The network is constructed through an approximation argument with no quantitative width budget and is not identified with the model returned by training. Moreover, the auxiliary function φ\varphi, its norm, and the interpolation region can change with the selected data. Consequently, the displayed geometric factors alone do not establish an error improvement when comparing subsets at fixed model capacity. Since the paper’s central contribution is an explanation of data efficiency, these missing connections materially weaken its theoretical contribution.
  2. The experimental protocol does not adequately separate geometric optimization benefits from changes in the training objective. The main LLM efficiency metric can be written as Tτ(S)=inf⁡{t:L^S(θS(t))≤τ}T_\tau(S)=\inf\{t:\widehat L_S(\theta_S(t))\le\tau\}, where L^S\widehat L_S is the training loss on the selected subset SS. Although the threshold τ\tau is shared, different subsets define different objectives. A smaller Tτ(S)T_\tau(S) can reflect easier examples, a lower initial loss, or changes in task composition, without demonstrating improved conditioning. Matching example counts also does not ensure comparable token exposure. Evaluation on common downstream benchmarks is useful, but evaluating only at these subset-specific loss checkpoints does not establish which method achieves better quality under the same training budget. Comparisons of common held-out performance against training time or processed tokens, together with controls for initial difficulty and data composition, are needed to substantiate the claimed mechanism. The reliability of the smaller performance gains is also unclear because variability across selection and training seeds is not reported. For example, the WizardLM result is 43.82% versus 43.63% for full-data training, while Tulu reports 50.04% versus 49.83% for equal-size random selection. These observations are promising, but without uncertainty estimates and comparable-budget learning curves, I cannot determine how robustly they support the claimed advantage.
Questions for authors
  1. When comparing subsets, how are model capacity, the integration domain I∘I^\circ, and ∥φ∥Wm,p\|\varphi\|_{W^{m,p}} controlled? What conclusion about models learned from the selected observations follows from Theorems 3–4 after accounting for these dependencies?

  2. Is there direct evidence that farthest-first selection improves the relevant Jacobian geometry—for example, the residual-aligned quantity r⊤JJ⊤r∥r∥2\frac{r^\top JJ^\top r}{\|r\|^2}, and that this improvement predicts faster loss reduction? Evidence from the controlled regression setting would materially strengthen the connection between the theoretical analysis and the observed benefits.

  3. Do the downstream gains persist at matched training-time or processed-token budgets, across multiple selection and training seeds? Please also clarify whether subsets differ in initial training loss, sequence length, or task composition, and provide uncertainty estimates for the smaller accuracy differences. These comparisons would affect my assessment of both the practical benefit and its attribution to geometry.

Reviewer_L26fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how data geometry affects the learning signals for LLMs. The paper argues from two theoretical views: optimization perspective (nearly identical inputs yield nearly dependent Jacobian rows) and approximation perspective (a data-dependent Bramble–Hilbert lemma shows the error bound depends on local diameter). Across regression on various datasets and models, the paper shows that data subsets selected to be geometrically uniform via farthest-first traversal in a fixed embedding space train faster than equal-size random subsets.

Critical strengths

Data scaling is important due to data scarcity. The paper’s takeaway is that effective data scaling means expanding coverage while avoiding redundancy and poorly conditioned geometry.

Critical weaknesses

The “geometry” in the paper seems to be a mix of fixed, model-independent representation space and model-dependent space (embedding or gradient space). For example, in some experiments the paper's method uses Word2Vec sentence embedding; on the other hand, the optimization view is built on gradient space. Can authors comment about how the choice of "data feature space" affects the paper's conclusions or recommendations?

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.