acceptodds
Under review as a conference paper at ICLR 2027

One-Hot Is Already a Geometry: What a Label Space Buys, in Bits

Abstract

One-hot targets are a choice of geometry: equidistant points, the regular simplex of neural collapse. Nothing forces that equidistance, and since labels a classifier already confuses could sit close together, placing classes at informative distances is common. But no unit says what that buys, or whether the gain comes from where the points lie or from which label sits where. We define one, in bits. With a frozen encoder and a ridge map onto the label points, scored by distance, the held-out information gain of a label geometry splits exactly into a reference level, an arrangement gain and an assignment gain, each against its own null. Three things follow. Averaging any geometry over all its labellings returns the regular simplex, so one-hot is the non-informative prior over assignment: at fixed temperature it arranges as well as any geometry of the same scale, and assigns nothing at all. An informative geometry trades arrangement for the power to assign. Near the simplex the arrangement gain depends on just two numbers read off the label Gram matrix, its distance from the simplex, and how much of that distance lies on the radii. A second-order expansion weighs them, and although it misses the level, only the ratio of its two weights fixes the order. Both weights come in closed form from the ridge, so screening a candidate geometry is a matrix product, not a training run. Applied to affective and topic classification, those two numbers order candidate geometries where the log-determinant, blind to the radii, fails.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Label geometry, V-information, Representation Evaluation, Permutation Null

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JvrpOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper views one-hot labels as a form of label geometry in the shape of a regular simplex, where all classes are actually located at the same distance. The authors decompose the information gain of label geometry into three components—reference level, arrangement gain, and assignment gain—and measure, on a per-bit basis, how much the arrangement of the points itself and the actual label-to-point correspondence each contribute. Theoretically, one-hot encoding contains no assignment information at all; however, under fixed conditions, it maximizes arrangement gain among all possible geometries, while meaningful geometries gain assignment gain by sacrificing a portion of that arrangement gain. In experiments on sentiment classification, label-name geometry lost arrangement performance compared to one-hot encoding but gained greater assignment gain, resulting in higher total information gain for some neural encoders.

Critical strengths
  1. Beyond the idea of “placing semantically similar classes close together”, which has been addressed in previous label embedding research—this study decomposes that effect into reference level, arrangement gain, and assignment gain, enabling an analysis of the causes behind the performance improvements.
  2. By interpreting one-hot encoding not as a simple baseline encoding but as a label geometry in the form of a regular simplex, and demonstrating that averaging all label assignments results in a return to this simplex, the study formalized one-hot encoding as a non-informative reference point for assignment.
  3. It theoretically explains that semantic label geometry creates a trade-off—gaining assignment information at the cost of some sacrifice in arrangement performance, and quantifies this as arrangement rent and assignment gain, thereby enabling an analysis of when this approach is more advantageous than one-hot encoding.
Critical weaknesses
  1. Since the core idea semantic label geometry itself overlaps significantly with existing research on label embedding, prototype, and taxonomy-based classification, the practical novelty of this paper lies primarily in its decomposition and interpretive framework rather than in a new representation method.
  2. The main theoretical results hold most strongly under the restricted setting of a frozen encoder and a ridge decoder; however, since the same trade-off is not maintained with a more flexible decoder, generalization to real-world deep learning settings remains unclear.
  3. While the proposed second-order predictor explains the relative rankings of candidate geometries to some extent, it fails to accurately predict the absolute level of gain when the actual published geometry is not sufficiently close to the simplex, indicating limitations in its practical predictive power.
  4. The experiments focused on a limited dataset centered on sentiment classification and a small number of encoder combinations; therefore, it has not been sufficiently verified whether the same decomposition and trade-off hold true for problems with a much larger number of classes or strong hierarchical structures, as well as for vision classification and large-scale label spaces.
Questions for authors

Are there any benchmarks that use one-hot encoding in modern NLP tasks? If so, does the explicit geometric regression-to-label-space setting analyzed in this paper align with the standard end-to-end classification pipeline?

Reviewer_paTiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether meaningful label embeddings instead of traditional one-hot labels can improve classification. It proposes a framework to separate the benefits of how label embeddings are arranged from the benefits of assigning the correct labels to them. The paper outlines some theoretical results to show that one-hot encoding provides an optimal geometric arrangement under certain assumptions, but meaningful label embeddings can still perform better by capturing relationships between classes. They also propose a simple measure to predict the usefulness of a label geometry.

Critical strengths

a. Novel theoretical contribution: The paper provides a useful framework for separating the benefits of label geometry from the benefits of assigning meaningful labels to the embeddings. b. Strong experimental analysis: The authors evaluate different label geometries and decoders. They also have experiments that highlight the limitations of their approach.

Critical weaknesses

a. Beyond Ridge Regression: The results look good for Ridge Regression but not so much for a regular neural network decoder. How useful and generalizable is the proposed approach. b. Theoretical assumptions: The results reported in the paper are for a specific decoder and temperature. How will this extend to end-end- classifier models.

Questions for authors

a. Generalizability: The benefits of semantic label embeddings seem to disappear when considering a more flexible neural network decoder. How applicable are the theoretical findings to modern end-to-end classification models? b. Practical significance: Under what conditions should researchers use semantic label embeddings over conventional one-hot labels?

Reviewer_UKseOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper is on quantifying information gain afforded by a label representation (for a fixed encoder). At a high level, the paper proposes a certain quantification (which is called a "unit") of this information gain, observes that the one-hot representation (equivalently, the simplex) is a natural notion of null in terms of label representations. It further decomposes its definition of information gain into two parts: the gain (or really loss) arising from deviating in arrangement from the simplex, and the gain from assigning explicit labels to the elements of a frame (which, e.g., may be organized so that semantically similar labels occupy similar label vectors).

\newcommand{\ig}{\mathrm{IG}} \newcommand{ar}{\operatorname{arrGain}} \newcommand{\as}{\operatorname{asgnGain}} \newcommand{\softmax}{\operatorname{softmax}} \newcommand{\nll}{\mathrm{NLL}}.

The quantification proposed arises thus: suppose that one is trying to predict a label using ridge regression: herein, one may regress (encoded) features X onto a given representation of the labels. Observe that because ridge predictions are linear in the response observations, one can decompose the effect of a different label representation linearly. Now, for a given training dataset (X,Y), validation set (\tilde{X}, \tilde{Y}), let GG be the operator that computes first regresses X onto Y (in the one-hot simplex representation), and then produces the predictions for Y~.\tilde{Y}. Then Y^\hat{Y} under the label representation ZZ is simply GZGZ.

Now, of course, this ridge procedure does not itself produce a label, or even a distribution over labels. The authors consider the following way to do this: for a ridge prediction y^,\hat{y}, let DℓD_\ell denote the squared distance of y^\hat{y} for zℓz_\ell, the representation for the ℓ\ellth point, and let p(y)=\softmax(−D/τ)p(y) = \softmax( -D/\tau) be the induced distribution. For any representation ZZ, its associated normalised log likelihood for label ii is \nlli:=E[−log⁡p(i)∣y=i],\nll_i := \mathbb{E}[ - \log p(i) | y = i], where the average is occurring over draws of (validation) instances, and the information gain is $ \ig(Z) := \frac{1}{n} \sum_i (\log (n) - \nll_i),$ where (in a roundabout way in the text) all the logs are base 2. Evidently, this can be measured using a validation set. This information gain is presented as a measure of the quality of the representation. It should be noted that the distribution pp depends on ZZ through only the Gram matrix K=ZZ⊤,K = ZZ^\top, and so we can equivalently identify these measures as functions of KK. I'll use both below, as the paper does. In any case, the paper standardises KK by demanding that its rows (and thus columns too) are centred, and that its trace is fixed to ndnd.

The authors further propose a decomposition of this \ig\ig. This arises in two steps, stemming from the fact that one can see a label representation as jointly specifying an arrangement of points to the label space, and an assignment of specific label identities to each of these points. Note that the latter property is nontrivial, in that in problems with semantic structure on the labels (or I guess in hierarchical problems too, although the paper doesn't note this), there are certain senses in which subsets of labels are more naturally grouped together, and even for a given arrangement, a poor assignment can violate such implicit structure. Note specifically that one can modify assignment by permuting the entries of ZZ. The authors propose explicitly capturing what each of these aspects gives thus:

  • First a reference level is defined. This arises as M0:=EZˉ[\ig(Zˉ)]M_0 := \mathbb{E}_{\bar Z}[\ig(\bar{Z})], where Zˉ\bar{Z} is drawn thus: draw nn iid points according to a reference distribution (Gaussian) appropriately normalised, and permute it arbitrarily (for iid reference, the permutation is unnecessary). This M0M_0 is a reference property of the learning task at hand.

  • Second, at an "arrangement level", the arrangement of ZZ is fixed, but the assignment is randomised, giving M1(Z)=Eπ[\ig(PπZ)],M_1(Z) = \mathbb{E}_{\pi} [\ig( P_\pi Z)], where π\pi is drawn uniformly over permutations, and PπP_\pi is the standard matrix representation of π\pi.

  • Third, at the arrangement and assignment level, both values are fixed to get M2(Z)=\ig(Z)M_2(Z) = \ig(Z).

Then the effect of the arrangement is isolated as \ar(Z):=M1(Z)−M0(Z)\ar(Z) := M_1(Z) - M_0(Z) and that of assignment with a given arrangement as \as(Z):=\ig(Z)−M1(Z)\as(Z) := \ig(Z) - M_1(Z).

An observation is then made that the arrangement gain can only be smaller than that for the simplex: is because \ig\ig is a concave map of the Gramian KK, and the average gramian under the permutation structure is that of the simplex, denoted KsK_s, giving \ar(K)≤\ig(Ks)−M0.\ar(K) \le \ig(K_s) - M_0. Further, it is noted that the assignment gain of the simplex is nought, because permuting the simplex leaves it invariant. In other words, M1(Ks)=\ig(Ks)M_1(K_s) = \ig(K_s). This sets up the simplex as a notion of null: it maximises arrangement gain, and has no assignment gain, and further arises naturally from the simple priors picked.

This sets up a notion of "arrangement rent," defined as \ig(Ks)−M1(K)=M1(Ks)−M1(K)\ig(K_s) - M_1(K) = M_1(K_s) - M_1(K), the amount of information that choosing the non-flat arrangement costs. This needs to be offset by the assignment gain in order for the label representation to gain, in terms of \ig\ig, over the flat representation.

Nex, the authors work out that in the limit as ∣K−Ks∣→0\\| K - K_s \\| \to 0, the arrangement gain can be written as a linear function of the fluctuation in the diagonal entries of KK, and the trace of K2K^2 up to the third order, which arises from symmetry considerations induced by the uniform permutation, and an explicit formula is developed in the appendix (specifically, coefficients of these two entries are derived in terms of the ridge fit in a certain limit). It is argued that this captures arrangement gain better than only a spectrum-dependent measure like logsumdet.

Finally, the paper computes these three metrics for four choices of label representation (the simplex, one "label-name representation" arising as an average of many text embeddings, and two from psychometrics), which are evaluated against four encoders (3 neural, one non-neural). The main observation appears to be that the label-name rep indeed gains over the simplex in terms of \ig\ig, and that this (necessarily) occurs because the assignment gain wins over the arrangement rent. Further, evaluation of the functional capturing the the arrangement gain is given, and in particular both it and a radial-index (essentially the variance of diagonal entries term above) capture it much better than purely spectral quantities.

Critical strengths

The paper is certainly on topic, and studies an ambitious problem. I checked most of the arguments underlying technical results, and they are correct. Certain aspects of this paper are quite interesting. For instance, while it is intuitive that choosing any arrangement that is not equiangular and equal norm will cause the squared distance based rounding procedure to lose something on average, since some label representations are now overcrowded, the explicit decomposition through the two notions of gain, and the bound relative to the simplex, is a nice way to express this, and to underscore that deviation from equiangularity can only help through how (semantic) labels are distributed over the frame.

Critical weaknesses

there are important limitations to this paper, which determine most of my rating below. These are to do with the lack of any operational contextualisation of \ig\ig and with certain choices in defining it which render it far from canonical in any sense, and to do with the writing.

  • On the information gain. Ultimately IG is a metric proposed for the quality of label representations. To this end, it is natural to ask what one needs from a metric (or, if you like, a "unit") in order for it to be useful. There are two important criteria: either the metric must be capturing some fundamental property in a canonical way, or it must have operational pertinence by either characterising certain limits to the problem, or by enabling comparisons between methods in a way that was unavailable without them. The paper does not discuss either of these aspects, and in fact in my opinion the metric IG certainly misses on the first ground, and it is unclear that it can deliver something of the second sort.

    • Is IG canonical in some sense? There are two nontrivial ingredients in the IG metric: a) the fact that one uses a ridge regression solution to map from features to a mixed estimate, and b) the fact that this is mapped to labels using a Gibbs law with the (negative) square-difference of the estimate from the individual label representations. Both of these steps seem quite arbitrary. Why should I use a "mixed prediction" that is a linear map of the input encodings, and even if we do insist on this, why I should use ridge over some other linear regression method. For instance, the most natural thing is to directly optimise the cross entropy of the prediction laws I'm going to use. Further, it is not obvious why I should use the squared distance as the functional to take softmax over: I could use distance, cosine similarity, some sort of Mahalanobis metric, whatever. Given this, it is unclear why one should build a metric out of one particular way of doing things. The main plus I see is that some of the subsequent maths in the paper is enabled by the fact that the resulting log-probabilities are concave functions of KK, but this is hardly a reason to build a generic unit for the value of a label embedding out of.

    • The operational role of IG: The paper doesn't tell the reader what a concrete application of IG is. In other words, okay, given a certain label representation, I can now compute \ig\ig for a given problem, but what should I do with it? Is it the case that if \ig\ig is big for one problem, it will remain big for another similar problem, in other words, is the application some sort of transfer? Or is the idea that \ig(K)\ig(K) may admit efficient maximisation, so the setup of the theory gives one a good way to arrive at a good label representation for a given dataset, encoder, and task? Or something else? Of course, any such proposed application must be accompanied by supporting analysis (if transfer, one must demonstrate some sort of accuracy in it, if optimisation, one must show that the gain in IG is nontrivial, and that this improves performance on the downstream task, along with somehow dealing with the choice of τ\tau). Without something like this, the metric remains an abstract object without operational pertinence.

    Without these considerations, in my opinion, the principal contribution of the paper doesn't hold water. At best, the paper is not ready for publication, and significant more work is needed.

  • In my opinion, the writing of the paper is, frankly, terrible. Things are written in an overwrought style which obscures even simple observations, to the degree that I had to repeatedly re-read paragraphs just to figure out what statements were being claimed, never mind evaluate them or their reasoning. This is further not helped by the fact that paragraphs have very poor delineation between definition, commentary, derivation, and details for experiments that will be presented only pages later. Further, even when given, important technical details are discussed in an opaque way: consider for instance the definition of the canonical temperature on line 147, for which I'm still not sure if this temperature depends on the validation set used to compute \ig\ig or not. Note that I am not talking about the style per se, only about clarity of definitions, claims, and evidence, and the delineation of these classes of statements from commentary.

    • Let me give just one example to justify this opinion. Take, e.g., the paragraph from line 197 to 208, which set up the principal objects of arrangment and assignment gain. Even before discussing the merits of the decomposition, an experimental detail of how the identity (4) is explicitly checked before computing tables is mentioned, distracting from the point (btw, a = c + (b-c) + (a-b) is not something one needs to computationally check :) ). The justification for the pieces, namely that each piece is computing an increment in infomration gain upon revealing further information about the label representation, is shoved into the cryptic term "each part is a difference against a declared null". Then we get a bunch of tautologies: each part has a sign "and a test" (which I don't know what having a test means), and that they need not be positive. Then an interpretation of these values being negative is given, but even this is confusing: for instance, for the first time the term "pairing" is introduced (without explanation) for the assignment of particular labels to vectors in ZZ. Strangely, this paragraph about introducing the concept is then followed (210-216) by an experimental detail about how these quantities are estimated in section 4, and what error bars are deemed meaningful, which should all appear much later. Even this paragraph is confusing, e.g., when describing resolution, all that is meant is that a 3 sigma bound is used for significance, but even this simple thing is said in a cryptic two line statement.

    None of this is impossible to understand, but it is also not easy to understand. This is pretty much the case for the whole paper, and the experience of reading it was thus singularly unpleasurable to me.

  • A relatively minor weakness is that I was totally lost on the point of the theory involving Vˉ\bar{V}. This matters because this stuff occupies a significant chunk of the paper.

    • First, I understand that this is presented as a proxy for arrangement gain, but the metric under study is the whole information gain \ig\ig, right? As far as I can tell, Vˉ\bar{V} does not shed any light on the assignment gain, and thus on the infomration gain. So how should one use it? Or equivalently, what can one do with an asymptotic proxy for the arrangement gain in isolation?
    • Much is made of the fact that Vˉ,\bar{V}, unlike the log-determinant, is sensitive to non-spectral information about the label representation, and that this is a good thing. But the thing is, I personally never thought that all the salient aspects of a label representation can be captured through its spectrum, so I don't know what to make of this fact. I understand the broad point that, given a fixed trace, a large determinant signals that the energy of the spectrum is well spread out, but it is unclear why this should determine how a given encoding will interact with a given frame in which labels are placed. Perhaps this view is prevalent in the literature (which is not really discussed in the main text at all), but then appropriate references should be cited at the point where this contrast is raised, which is totally absent!
    • Of course, this extends to the experimentation: all of the experiments which show that Vˉ\bar{V} is indeed predictive of the assignment gain are a bit moot until I understand what I can do with it in isolation.
Questions for authors
  • In the ridge term in Fig. 2, you are (rightly) sensitive to the frequencies of the various labels in the corpus. But then I wonder why this doesn't show up in the definition of \ig\ig? In other words, why is it that \ig\ig should be a flat average over the various labels, and not a weighted average in terms of the frequency (and similarly, why log⁡(n)\log(n) should enter instead of −log⁡(freq(i))-\log(\mathrm{freq}(i)))?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.