- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper is about "embeddings" and the choice of their dimension mostly in classification tasks. The main concern of the paper are the rules of the form n > c ln N. The Authors provide an experimental study based on linear regression and synthetic data.
Critical strengths
- The topic of identifying the correct number of dimensions while using embedders is under-developed and highly relevant to ICLR community
Critical weaknesses
-
In my opinion the contribution is incremental -- the paper does not feel like a proper study, instead it looks like a working report from a numerical exercise. Moreover, the disclosed usage of AI agent raises serious concerns about the actual contribution of the human authors -- AI agent was involved in nearly each crucial step of the study
-
The presentation is really awful -- the paper lacks the definitions (we are never told what embeddings we consider -- and machine learning has at least two communities that use that word), correct problem statement, the motivation. It is unclear who should be interested in those results, Most of the notation is left without explanation which is a major flaw in mathematical research. Incorrect usage of wording, e.g., paper mentions "metrics" in their main "protocol" while discussing, e.g., accuracy which is not a metric (it is a measure -- not every measure is a metric!). Even if the paper feels like a numerical report, most of the justification is left without explanation (e.g., no diagnostics of the linear model that is used in a way aimed at explanation; strange statements on the sign being the "property" of estimator -- we cannot say something like that)
-
The proposed usage of linear regression is strange -- on one hand it looks like something trivial, but at the same time it seems incorrectly used/performed (I lack some details to say for sure). The results in Table 3 suggest lack of significance, but without the proper diagnostics (we are not given the details even if those regressions are suggested as the contribution!) we should not comment on that. The presentation of this part of the paper is obfuscated -- I am unsure if I understood correctly the dependent variable (if that is the slope, the whole procedure seems incorrect as it forms some sort of the linear regression on the time series?)
Questions for authors
I do not think that my final score can be increased, but I really would like to get some explanations, e.g.:
- The numbers in Table 3 are not the same as those reported in Section 3.4 -- I do not know where numbers from that section come from?
- provide more information on the setup of the regression (the exact functional form, the log from the procedure)
Response to Reviewer 8ksP: where the Section 3.4 numbers come from, and the exact regression
Q1. Table 3 and Section 3.4 report different pipelines. You are right that the text did not label them. Section 3.4 starts from a pipeline with an unshuffled split and removes its problems one by one; Table 3 is the final pipeline. The revised PDF adds Table 4, which lists all four cases at d=8 (pooled OLS, 95% t-intervals):
While rechecking Section 3.4 we found an error of ours: the median realised excess of the maximum over acc(128) (0.29) had been computed on the shuffled-split cells, as the upper of the two middle values. Section 3.4 now gives it for the unshuffled split the paragraph describes: median 0.34, largest 1.62 (Appendix C records the correction).
Q2. The exact regression. The dependent variable is not a slope, and there is no time index. The unit is one graph cell, a dataset at one subgraph size: 12 cells (Cora 3, CiteSeer 4, PubMed 5). For cell c and width d,
yc = mean over 5 seeds of [acc(128) - acc(d)], in accuracy points (test accuracy at the best validation epoch),
and yc=a+blnNc+ec is fitted by OLS; b is the reported slope. The fixed-effects column adds one intercept per dataset; the cluster-robust column uses CR1 errors by dataset with G−1 degrees of freedom (G=3), because cells of one dataset are nested subsamples. Section 2.1 now states this form.
Diagnostics at d=8, now in Appendix B with all 12 values of yc: a Shapiro-Wilk test does not reject normal residuals (W = 0.94, p = 0.51); the squared residuals fall with ln N (p = 0.011), as the seed noise does (Section 3.4), and with heteroskedasticity-robust HC3 errors the slope is -0.80 [-2.32, +0.72] (p = 0.27); the most influential cell is CiteSeer at N = 399 (Cook's distance 1.07, leverage 0.26), and without it the slope is -0.22 [-1.02, +0.59]. Read as a minimum, the rule (49.9 to 82 dimensions on these cells) implies a gap above 5% of acc(128) at d=8 in every cell. The gap is 0.3 to 7.2 points, exceeds that bar in 2 of 12 cells, does not grow with N, and every 95% floor is 2-32.
Presentation. We accept these points, and the revised PDF (changes in blue) makes them:
AI use. The statement lists every category of the ICLR 2027 policy that applied, and how the work was checked. The authors reviewed all of it and take responsibility for the content. Appendix C is now titled "Corrections to earlier analyses".
Finally, on scope: besides the regressions and the synthetic word vectors, the paper measures real data (text8 word vectors, an attention head trained on associative recall, and GCNs on Cora, CiteSeer and PubMed).