- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how data geometry affects training efficiency and approximation error. The authors focus on data uniformity and argue that a smaller and well-distributed dataset can perform better than a larger dataset with redundant samples. The paper develop two theoretical results. The first relates the distance between training samples to the smallest eigenvalue of the Jacobian Gram matrix. The second studies approximation error using local simplex geometry. The authors consider a farthest-first data selection method. The numerical results show that uniform subsets can train faster and achieve similar or better performance than larger datasets.
Critical strengths
The paper studies an important problem. Using more training data does not always improve learning efficiency. The authors try to explain this through both optimization and approximation theory; see Subsections 3.1 and 3.2, pp. 5-6. I find this connection interesting. The discussion of the trade-off between data separation and coverage is also useful.
Critical weaknesses
I have two main concerns.
The first concern is the theoretical result. (i) Theorem 2 (p. 5) gives an upper bound on the smallest eigenvalue of the Jacobian Gram matrix. However, this does not show that increasing the minimum distance between samples will improve convergence. The authors also acknowledge this in the paper but did not deal with it. (ii) Theorem 3 (p. 6) shows that the approximation bound depends on the triangulation and an auxiliary function. But these quantities will change across datasets. Thus, it is difficult to use this result to explain why one selected subset should outperform another.
The second concern is the real performance. If we pay attention to Figure 4 panel (b) on page 8, the results show 43.82% for uniform selection, 43.09% for random selection, and 43.63% for the full dataset. I am not convinced that such small difference matters in practice. The authors report improvements in training time, which is quite obvious given different sample sizes. However, I am wondering whether reaching the same training loss is a fair comparison when the datasets contain different samples.
Questions for authors
My questions will be based on my comments for critical weaknesses.
(1) How does Theorem 2 support the claim that more uniform data lead to faster convergence? It would be great if the authors can provide some numerical evidence showing that uniform selection improves the relevant eigenvalues during training.
(2) Since the approximation bound in Theorem 3 depends on quantities that could change with the selected dataset. how can we use this bound to compare different subsets? I am specifically interested in understanding how this result relates to the neural networks trained in the experiments.
(3) I am confused how the authors calculate the MSE, MAE etc. in the paper. I am not sure whether the numerical results are averaged over multiple runs. Also, was the stopping criterion in Subsection 3.3 used to determine the subset sizes in the experiments?
(4) Figure 1 is obviously generated from AI (correct me if I am wrong). Figure 4 (b) and Figure 5 (b), the legend is out of the figure box. There is no journal details for the citation "Krzysztof Kurdyka and Laurentiu Paunescu. Hyperbolic polynomials and multiparameter realanalytic perturbation theory. 2008." I understand that the authors acknowledge the use of AI for language editing. But please recheck the paper and ensure figures/writings are aligned with academic standards.