acceptodds
Under review as a conference paper at ICLR 2027

How Many Dimensions Do You Actually Need? Testing Embedding-Dimension Rules on Their Slope, Not Their Value

Abstract

Rules of the form , where counts the items to be represented, are widely used to choose an embedding dimension. They are usually validated by evaluating the rule at one and comparing it with a conventional width, a check that a wide family of constants and functional forms passes equally well. In this work, we measure how the required dimension actually moves with , holding the task fixed and varying over up to decades in associative recall with an attention head, identity retrieval and topic recovery in word embeddings, and node classification on citation graphs. We find that the dimension a task needs is set by what it must distinguish rather than by . On the same synthetic embeddings with few planted topics, recovering the topics needs as many dimensions as there are topics whatever is, while telling the words apart needs a width that grows with . We also find that identity tasks on different data grow at rates that differ by more than an order of magnitude, and that classification tasks show no detectable common growth. Along the way, we show that dimension floors estimated relative to a noisy curve's maximum acquire a spurious slope in , and give an estimator that does not. We hope the protocol lets future dimension rules be stated for a task and tested on their slope.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
embedding dimension, dimension selection, scaling with N, word embeddings, attention head size, graph neural networks, dense retrieval, minimum entropy principle, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8ksPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about "embeddings" and the choice of their dimension mostly in classification tasks. The main concern of the paper are the rules of the form n > c ln N. The Authors provide an experimental study based on linear regression and synthetic data.

Critical strengths
  • The topic of identifying the correct number of dimensions while using embedders is under-developed and highly relevant to ICLR community
Critical weaknesses
  • In my opinion the contribution is incremental -- the paper does not feel like a proper study, instead it looks like a working report from a numerical exercise. Moreover, the disclosed usage of AI agent raises serious concerns about the actual contribution of the human authors -- AI agent was involved in nearly each crucial step of the study

  • The presentation is really awful -- the paper lacks the definitions (we are never told what embeddings we consider -- and machine learning has at least two communities that use that word), correct problem statement, the motivation. It is unclear who should be interested in those results, Most of the notation is left without explanation which is a major flaw in mathematical research. Incorrect usage of wording, e.g., paper mentions "metrics" in their main "protocol" while discussing, e.g., accuracy which is not a metric (it is a measure -- not every measure is a metric!). Even if the paper feels like a numerical report, most of the justification is left without explanation (e.g., no diagnostics of the linear model that is used in a way aimed at explanation; strange statements on the sign being the "property" of estimator -- we cannot say something like that)

  • The proposed usage of linear regression is strange -- on one hand it looks like something trivial, but at the same time it seems incorrectly used/performed (I lack some details to say for sure). The results in Table 3 suggest lack of significance, but without the proper diagnostics (we are not given the details even if those regressions are suggested as the contribution!) we should not comment on that. The presentation of this part of the paper is obfuscated -- I am unsure if I understood correctly the dependent variable (if that is the slope, the whole procedure seems incorrect as it forms some sort of the linear regression on the time series?)

Questions for authors

I do not think that my final score can be increased, but I really would like to get some explanations, e.g.:

  • The numbers in Table 3 are not the same as those reported in Section 3.4 -- I do not know where numbers from that section come from?
  • provide more information on the setup of the regression (the exact functional form, the log from the procedure)
1 comment
AuthorsOpenReview ↗

Response to Reviewer 8ksP: where the Section 3.4 numbers come from, and the exact regression

Q1. Table 3 and Section 3.4 report different pipelines. You are right that the text did not label them. Section 3.4 starts from a pipeline with an unshuffled split and removes its problems one by one; Table 3 is the final pipeline. The revised PDF adds Table 4, which lists all four cases at d=8d = 8 (pooled OLS, 95% t-intervals):

  • unshuffled split (validation and test cut from an unshuffled remainder), gap against the curve's own maximum: -1.14 [-1.91, -0.37], p = 0.008
  • unshuffled split, gap against the fixed reference acc(128): -1.06 [-1.90, -0.22]; with dataset fixed effects -0.74 (p = 0.090), cluster-robust p = 0.128
  • shuffled split, gap against the maximum: -0.95 [-1.90, +0.00]
  • shuffled split, fixed reference, i.e. Table 3: -0.80 [-1.89, +0.28]

While rechecking Section 3.4 we found an error of ours: the median realised excess of the maximum over acc(128) (0.29) had been computed on the shuffled-split cells, as the upper of the two middle values. Section 3.4 now gives it for the unshuffled split the paragraph describes: median 0.34, largest 1.62 (Appendix C records the correction).

Q2. The exact regression. The dependent variable is not a slope, and there is no time index. The unit is one graph cell, a dataset at one subgraph size: 12 cells (Cora 3, CiteSeer 4, PubMed 5). For cell cc and width dd,

ycy_c = mean over 5 seeds of [acc(128) - acc(d)], in accuracy points (test accuracy at the best validation epoch),

and yc=a+bln⁡Nc+ecy_c = a + b \ln N_c + e_c is fitted by OLS; bb is the reported slope. The fixed-effects column adds one intercept per dataset; the cluster-robust column uses CR1 errors by dataset with G−1G - 1 degrees of freedom (G=3G = 3), because cells of one dataset are nested subsamples. Section 2.1 now states this form.

Diagnostics at d=8d = 8, now in Appendix B with all 12 values of ycy_c: a Shapiro-Wilk test does not reject normal residuals (W = 0.94, p = 0.51); the squared residuals fall with ln N (p = 0.011), as the seed noise does (Section 3.4), and with heteroskedasticity-robust HC3 errors the slope is -0.80 [-2.32, +0.72] (p = 0.27); the most influential cell is CiteSeer at N = 399 (Cook's distance 1.07, leverage 0.26), and without it the slope is -0.22 [-1.02, +0.59]. Read as a minimum, the rule (49.9 to 82 dimensions on these cells) implies a gap above 5% of acc(128) at d=8d = 8 in every cell. The gap is 0.3 to 7.2 points, exceeds that bar in 2 of 12 cells, does not grow with N, and every 95% floor is 2-32.

Presentation. We accept these points, and the revised PDF (changes in blue) makes them:

  • Section 2 opens with a "Terms" paragraph: the N items (words, attention keys, graph nodes), the embedding as a learned vector per item, its width in each setting (SVD rank, query/key projection width, GCN hidden width), the task, the required width n*, and the reference width. It also says who the result is for: anyone choosing a width from N with such a rule, including the graph selectors that prescribe one this way.
  • "Metric" in the protocol box is now "performance measure".
  • The sentence that the slope's sign "is no longer a property of the estimator" meant that the negative sign is no longer produced by the estimator's bias. Section 3.4 now says that, and the title of Figure 4a no longer says the estimator "decides the sign".

AI use. The statement lists every category of the ICLR 2027 policy that applied, and how the work was checked. The authors reviewed all of it and take responsibility for the content. Appendix C is now titled "Corrections to earlier analyses".

Finally, on scope: besides the regressions and the synthetic word vectors, the paper measures real data (text8 word vectors, an attention head trained on associative recall, and GCNs on Cora, CiteSeer and PubMed).

Reviewer_nh2ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at rules of the form n > c ln N for choosing an embedding dimension (concretely Su's 8.33 ln N and the GNN dimension selectors derived from it) and makes the point that validating such a rule at a single N against a conventional width is not a test: any constant of the right order passes. Instead they fix the task and vary N, and measure the slope dn*/d ln N. They provide evidence using four experimental settings: i) synthetic word vectors with K planted topics, where the same vectors are scored on topic recovery and on identity retrieval; ii) PPMI+SVD vectors on text8 scored on identity retrieval, N from 500 to 16k; iii) a one-head associative recall task where the QK width is swept, N from 8 to 4096; iv) GCN node classification on induced subgraphs of Cora/CiteSeer/PubMed.

Their findings are that separate K categories need about K dimensions at every N on the synthetic data, and show no common trend with N on the three graphs; tasks that must tell N items apart grow with ln N, at rates that differ by a factor of ~40 between settings (+2.4 for attention, +103 for text8).

Critical strengths
  1. The slope argument is simple and right. A rule n > c ln N is a claim about how n* moves with N. A value check at one N cannot separate constants (6.3 vs 10.4 both land within 25% of 96 at N=1e5) or functional forms, and the paper shows that these rules have in fact only ever been checked that way. The protocol box in §2.1 is good and reusable.
  2. The same-embedding, two-task experiment (Fig. 2 / Fig. 5a) is a clean demonstration: identical vectors, identical N, two scoring functions, floors of 8 and 32. Together with the fixed-reference estimator, the flat-null simulations, t-intervals, cluster-robust SEs and the list of withdrawn claims in App. C, the empirical work is more careful than usual for this kind of paper.
Critical weaknesses
  1. The headline ("the task sets the dimension, not N") rests on thin real-data evidence. One strong real-data number comes from a somewhat questionable task. On the graphs the pooled slope of the floors has CI [-16, +5], which contains both 0 and 8.33. Hence, three graphs cannot tell "flat" from "the rule"; the paper says so, but Fig. 1 and the abstract do not read that way. The synthetic category floor at K is built in by construction (co-occurrence of rank K). The attention "identity" line is, by the authors' own account, an Adam threshold, since a width-2 head solves the task at any N. What is left on real data is text8, where the identity task is nearest-neighbour matching of the rank-n reconstruction of a PPMI row against the N full-rank rows. That measures how fast the PPMI spectrum decays, not anything word vectors are used for.
  2. The functional form undercuts the framing. The paper's own AICc comparison (App. B) prefers a power law over a + b ln N on both identity tasks. If n* grows as N^alpha, "slope per unit of ln N" is not a constant and the quoted slopes (+2.43, +102.9) are averages over the chosen N range rather than properties of the task. The test as stated is then not quite the right test, and the paper only tests one rule (Su's); PIP loss, SNML etc. are discussed but not run. Smaller point: the estimator-bias contribution is weaker than §3.4 suggests. By their own numbers replacing the max by acc(128) moves the slope from -1.14 to -1.06; the bulk of the spurious slope came from the unshuffled split and pooling. Worth reporting, not a third contribution.
Questions for authors
  1. text8: why compare rank-n reconstructed rows to the full PPMI rows rather than rank-n to rank-n (i.e. retrieval in the embedding space)? And does any standard intrinsic evaluation at fixed vocabulary (similarity, analogy) show dependence of n* on N at all? If not, the "identity" result needs to be scoped much more narrowly.
  2. Given the AICc result, would you restate the test as estimating alpha in n* ~ N^alpha (log-log slope), and what do the rules predict in that parametrisation? Does the category/identity contrast survive?
  3. Have you related the floors to any intrinsic-dimension estimate of the data itself (PPMI rows at each N, node features, graph)? A hypothesis consistent with your results is that the identity floor tracks the intrinsic dimension of the item set rather than N; that would be a sharper statement than "the rate is set by the data" and would explain the factor of 42.
1 comment
AuthorsOpenReview ↗

Response to Reviewer nh2R: narrower headline, the log-log exponents, and the text8 spectrum

Thank you. We agree with W1 and with the main point of W2. The revised PDF makes the changes below (in blue).

W1. The headline was stronger than the real-data evidence. Agreed. The abstract, introduction, contributions, Figure 1 caption and conclusion now say that the task is isolated only in the synthetic experiment, where data, embedding and N are shared and only the scoring differs. On real data they claim less: the two identity tasks (one of them, attention, an optimisation threshold, as you note) add width per unit of ln N at rates a factor of 42 apart over their own N ranges, and the three graphs show no common growth.

On graphs, Section 3.3 now keeps two statements apart. The pooled floor slope, -5.53 [-16.24, +5.17], contains both 0 and 8.33, as you say, so three graphs cannot tell flat from the rule's slope. What they do test is the rule read as a minimum: it asks for 49.9 to 82 dimensions on these cells, while every 95% floor is 2-32. We agree the topic floor at K is partly built in (Section 4); the point of that experiment is that the same vectors' retrieval floor is not at K.

Q2. Log-log form. We now report it. Table 1 has the exponent α (slope of ln n* on ln N, t-intervals): text8 0.65 [0.35, 0.95]; attention 0.37 [0.32, 0.42]; synthetic retrieval 0.22 [0.13, 0.32] (K = 8), 0.23 [0.01, 0.44] (K = 16), 0.07 [-0.04, 0.18] (K = 32); synthetic topics exactly 0 for K = 8 and 16. A rule c ln N has local exponent 1/ln N whatever c is (0.16 at N = 500, 0.10 at N = 16,000); fitted over the same N as our series, its exponent is 0.21 on the attention grid and 0.13 on the text8 grid, and both measured exponents exclude it (for a rule with no intercept).

So the category/identity contrast survives on the same embeddings: topic exponent 0, retrieval exponent above 0 for K = 8 and 16, and at K = 32 the two do not separate, as before. Your point also changes how we read the factor of 42: it is a difference in width added per unit of ln N over two different N ranges, and in log-log form the two real exponents are closer (0.37 and 0.65, overlapping intervals). Section 3.2 now says this, and that the per-ln N slopes are averages over the N range, not constants of the task.

Q1. The text8 task. We compared against full rows because among rank-n vectors each word is its own nearest neighbour, so rank-n against rank-n is trivially perfect; the target has to come from outside the embedding. Using the full PPMI row, the text8 floor counts the spectral directions that keep the N rows apart; it does not measure a downstream use. Section 2.2 now says so, and adds that a larger N also brings in rarer words. We did not measure neighbourhood preservation (overlap of rank-n and full-space neighbours), or a fixed-vocabulary similarity or analogy score, against N; Section 4 lists both as untested.

Q3. Intrinsic dimension. We tested one version after submission, with the protocol and its reading fixed before the run. On the full text8 PPMI spectrum (the submitted records kept only its top 512 values), the rank holding 90% of the spectral energy grows slightly faster than N: 145 at N = 500, 3,469 at N = 8,000, exponent 1.15 [1.10, 1.20]. The floor's exponent is 0.65, and the floor falls from 0.44 to 0.10 of that rank, so it does not track the energy rank. A secondary measure, the participation ratio of the squared singular values, has exponent 0.48 [-0.03, 0.99], too wide to decide. Whether another intrinsic-dimension estimate tracks the floor is open (Section 4; the energy-rank result is in Section 3.2 and Appendix B).

W2, smaller point. Agreed. Replacing the maximum by acc(128) changes the slope only from -1.14 to -1.06; most of the apparent slope came from the unshuffled split and from pooling without dataset effects. The new Table 4 lists all four split-by-estimator cases, Section 3.4 now says so in its first paragraph, the estimator is no longer among the contributions (the pitfall is reported separately), and the conclusion says the fixed reference removes the bias, not the slope. Section 4 also states that only the entropy rule was tested; PIP loss and SNML were not run at each N.

Reviewer_wC1tOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how the required embedding dimension scales with the number of items (N). Rather than evaluating dimension-selection rules such as (n > 8.33\log N) at a single value of (N), the authors propose evaluating their slopes. Through experiments on word embeddings, attention-based associative recall, and graph node classification, they report that the required dimension depends on what the task must distinguish, rather than solely on (N). The paper also discusses estimation bias arising from using the maximum observed accuracy as a reference and proposes a fixed-reference estimator.

Critical strengths

S1. Potentially useful evaluation perspective.

The paper raises a reasonable concern about evaluating dimension-selection rules at a single value of (N). Examining how the required dimension scales with (N) could provide a useful perspective for empirical evaluation.

Critical weaknesses

W1. Limited interpretability of the experimental results.

The proposed protocol measures how the required dimension changes with NN, but the experiments do not sufficiently isolate the relevant factors. For example, varying NN also changes word frequencies in text8, sequence length in attention, and graph structure in node classification. Although the authors provide some controls, the results demonstrate setting-dependent scaling more convincingly than the central claim that the required dimension is determined by what the task must distinguish.

W2. Unclear novelty and significance of the proposed protocol.

The methodological contribution is not entirely clear. Evaluating scaling laws across multiple values of NN is a natural approach, and avoiding selection bias from noisy maxima is a well-known statistical consideration. Moreover, the 8.33log⁡N8.33\log N rule originates from a sufficient criterion, whose slope need not match that of the minimum required dimension. It remains unclear what additional insight or practical benefit the proposed protocol provides beyond standard empirical dimension sweeps.

Questions for authors

It would be nice if the authors can address the weakness above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer wC1t: what each experiment isolates, and what the slope test adds

Thank you. We agree with much of both points, and the revised PDF (changes in blue) narrows the claim to match.

W1. What isolates the task. We agree: only one experiment isolates "what the task must distinguish", the synthetic one, where data, embedding and N are shared and only the scoring differs (Table 1, Figures 2 and 5a). The abstract, introduction, contributions, Figure 1 caption and conclusion now say that, and describe the real-data results as what they are: setting-dependent rates for identity tasks, and no common growth on the three graphs.

On the factors that move with N:

  • Graph structure: a breadth-first sampler keeps mean degree flat in N (slope of ln degree on ln N +0.01, p = 0.859, against +0.67 for random subgraphs) and gives gap slopes of the same order, none positive and significant. It also changes the class mix, so the paper treats it as a second design rather than a degree-matched control (Section 4, Appendix B).
  • Sequence length in attention: N is both the length and the key vocabulary, and we do not separate them. At length 512, raising the vocabulary to 4096 moves the threshold from 8 to 11, a one-step change that other changes to the head also produce (Section 4, Appendix B).
  • Word frequencies in text8: the vocabulary is the N most frequent words, so a larger N adds rarer words. Section 2.2 now states this as part of what the text8 result measures.

W2. Novelty and significance. We agree that sweeping N is a natural design and that the selection bias of a noisy maximum is known (we cite Cawley and Talbot, 2010). The estimator is no longer among the contributions; the paper reports the bias separately as a pitfall (introduction, Section 4).

On sufficiency we agree, and Section 2.1 now says it directly: a sufficient width need not have the slope of the minimum, so the slope speaks to the rule's other two uses, as a minimum and as a prescription. Sufficiency is checked by the value at each N instead. For identity retrieval on text8, even the last failing grid width lies above the rule at 4 of 6 values of N at the 95% tolerance (5/6 at 98%, 6/6 at 99%; Section 3.2), so there the rule is not sufficient either.

What the slope test adds over a standard sweep: the rule is used as a prescription. The graph selectors built on it choose 98, 101 and 123 dimensions on graphs of 2.7k to 19.7k nodes, a rise that follows its ln N^2 term (Section 4). A prescription's content is its slope, and to our knowledge these rules had been validated by their value on each dataset, not on their slope (introduction, Section 2.1). The practical reading is now narrower than in the submitted version: on synthetic data with few categories the width can be sized from the number of categories, not from N; on the three graphs the floors show no common trend in N; and for identity tasks the rate differs by more than an order of magnitude between settings, over different N ranges, so it has to be measured on subsamples of the data at hand (Section 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.