acceptodds
Under review as a conference paper at ICLR 2027

Angular Density Represents an Embedding's Typicality

Abstract

Self-supervised pretraining has become the dominant paradigm for learning general-purpose representations across image, multi-modal, and language settings. Whether a given input is well-represented is normally assessed only indirectly, through labels, class structure, or expensive downstream evaluation. We show that an embedding's own geometry offers a cheap, label-free substitute: an embedding's *angular density* and, less consistently, its norm predict representation quality across paradigms. We argue that angular density measures an embedding's local *typicality* and does so even where no label- or class-based notion of typicality is available to begin with, e.g., individual LLM prompts or one side of a cross-modal image-text pair. Across vision, vision-language, and language models, embeddings with high angular density are more likely to be classified or answered correctly and, in vision and vision-language settings, represented similarly by other models. We additionally find that, in multi-modal models, the geometry of a *text* embedding predicts how reliably the corresponding *image* embedding is classified, despite no parameters being shared between the two encoders. We show that this local signal succeeds where a natural alternative, distance to a dataset centroid, does not: the global notion of typicality is inconsistent across models and often inverts, while angular density is monotonically predictive throughout.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self supervised learning, Angular Density, Typicality

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BGoDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores candidate definitions for measuring the “typicality” (as a proxy for quality) of an embedding, i.e., how difficult the corresponding input is for the model. Through a systematic comparison of supervised and unsupervised, global and local scores, across vision, language and multimodal settings, the authors identify angular density as the most consistent predictor of model quality among the candidates considered.

This geometric score outperforms supervised methods, with the benefit of working in cases where the notion of class or label isn’t available, but with the drawback of requiring enough statistical coverage of the latent space for meaningful inference.

They provide insightful links to known phenomena in latent geometry (e.g. neural collapse), as well as hypotheses on the mechanisms behind the emergence of the structure they leverage.

Critical strengths

S1 : Broad and systematic empirical evaluation. The paper evaluates a wide range of candidate geometric properties across vision, language, and multimodal representations. The comparisons span supervised and unsupervised measures, as well as global and local properties. They are not pushing for a method, they systematically compare many candidates and retain the best one. This makes the identification of angular density as a consistent predictor particularly convincing.

S2 : Clear problem formulation and positioning. The paper is well written and provides a clear overview of the problem and its relationship to existing work. It motivates the need for embedding-quality estimation in settings where label-based approaches are unavailable or unsuitable.

S3 : Useful methodological contribution. The introduction of the Property-Quality Correlation (PQC) metric provides a quantitative way to assess the relationship between geometric properties (e.g. angular density, distance to class centroid, etc …) and embedding quality (e.g. classification accuracy), accounting for both the strength and monotonicity of this relationship. This score is compared against more standard ones (e.g. Spearman’s ρ\rho or AUC). This score is not without degeneracies though, see W2.

S4 : Insights into representation geometry. The paper connects its findings to established phenomena such as neural collapse and explores possible explanations for the emergence of the geometric structure underlying angular density. These connections help situate the empirical results within a broader understanding of learned representations.

Critical weaknesses

W0 : Novelty. The authors cite Sun et al 2022 as a supervised metric requiring classes. This is not true. Sun et al.'s metric is exactly the same as the proposed one with the only difference being in the distance (euclidean for Sun and angular for this paper). The scope of both studies are related but different (detecting ood embeddings vs quantifying the quality of an embedding). I'd like the authors to give proper credit given how close these two works are.

W1 : Clarification on the notion of typicality. The abstract and introduction could better set the stage on typicality. I believe you are trying to say something like “we define typicality as …”, either with an abstract definition of which angular density is one operationalisation, or directly as angular density, and then mention that you also test other definitions but that angular density is the one you finally keep based on such and such analysis. Currently, the first mention of typicality is in the abstract : “We argue that angular density measures an embedding’s local typicality” (l16,17). At this point, this is either a definition of typicality, which is obscured by the use of “argue”, or a claim that angular density probes a notion of typicality yet to be defined. I believe you are in the former case, as I can’t find a definition of typicality that is not through a measure.

W2 : PQC degeneracies. PQC is well motivated but it looks hand crafted and has many very undesirable degeneracies. The most striking one is as follows. Take a property pp to be evaluated, and assume it almost exactly matches the quality qq you evaluate it with, say p=q=p = q = accuracy, except on a few number of points. Then PQC≤0.1\mathrm{PQC} \leq 0.1, and goes down with the number of bins. The second most undesirable property is the sensitivity of this score to the number of bins, which you do not appear to test. I’d argue that you were trying to craft a distributional distance, which could replace PQC without the trivial degeneracies. See Q2 for more detail.

minor weakness :

  • typo : figure 5 reports “knn accuracies” whereas I believe it should be “knn overlaps” in this context.
  • “Wang & Isola (2020) [...] with subsequent work [...] Liu et al., 2018.” : the 2018 ref can’t be subsequent work of the 2020 one.
Questions for authors

Q1 : Missing property ? Related to W0, I was wondering if you had tested euclidean density, and compared it to angular density. I can’t find a mention to this in the paper or appendices. For example, one could take the mean squared distances to the kNN, instead of the mean cosine.

For normalised embeddings, the two are equivalent, as ∥u−v∥22=2(1−cos⁡(u,v))\|u-v\|_2^2 = 2(1-\cos(u,v)) if uu and vv are unitary. In general though, it captures both the aspect of angle and norm, which you have separated in your study. Intuitively, this is the metric that makes the most sense, unless you argue (or show) that the norm information is unrelated to what you want to measure. In that case, discarding the norm is best. But from your own experiment, it seems that norm does carry some information.

Comparing to the distance to the kth NN in the main body (so Sun et al.) would strengthen the paper as it would be fully transparent and more anchored in the literature.

Q2 : Why PQC ? In the limitations, you mention that “Our per-bin PQC scores are moderate (0.2–0.6)”. I would argue that PQC is not very interpretable. There are many issues with PQC, e.g. the confound of whether the model is good at all. Due to the information term, a perfect model would give PQCs of 00. A mediocre model would have information terms relatively small. This is not exactly true though, since a good property would still categorize all successes in higher bins, so accuracy on D10 could significantly exceed the overall accuracy of the model. But the idea is there : you want the ranking given by pp to match the ranking given by qq. Here is the issue though.

A natural way to normalize PQC (or any other distance dd in general) would be to compare it to d∗=d(q,q)d^* = d(q,q) : the absolute ceiling of what you truly want to measure. This shows the second failure mode of PQC : if pp almost exactly matches qq, with qq the accuracy, the monotonicity term will be 0.1, meaning that PQC(q,q) = 0.1, even though there is a near perfect match between the property (pp) and the quality (qq). You can achieve that with q(D1)=ε,q(D2)=0,…,q(Di)=0,q(Di+1)=1,q(Di+2)=1−ε,…,q(D10)=1−εq(D_1) = \varepsilon, q(D_2) = 0, …, q(D_i) = 0, q(D_{i+1}) = 1, q(D_{i+2}) = 1 - \varepsilon, …, q(D_{10}) = 1 - \varepsilon.

Therefore, I’d argue that PQC should be replaced by histogram (or distributional) distances, weighing each sample with qq, and comparing the two orderings given by pp and qq gives two histograms, which can be as fine as you desire, and with the same total mass, so you can normalize them into densities, yielding a normalised, interpretable score.

Q3 : In the LLM case, do you have any intuition as to why instruct models are seemingly less organised than base models ?

3 comments

Dear Reviewer,

Thank you for a thoughtful review that will help us improve our work.

W0 (novelty, Sun et al.). You are right that we described Sun et al. as requiring class labels, and it does not. We will correct this and credit it prominently. Sun et al. score a test input by the Euclidean distance to its k-th nearest neighbour among L2-normalized penultimate-layer features of the training set, for OOD detection. The score uses no labels at test time, although their encoder is trained with labels (cross-entropy or SupCon). They also ablate an averaged-k distance, which is the closest variant to ours, and report that it performs on par with the k-th-NN distance (their Figure 3d). Ours is the mean cosine similarity to k neighbours, computed on mean-centered embeddings, to predict per-input reliability within distribution and across paradigms. The two are close, and we will say so. We have not compared the two scores directly, and we will state this.

W1 (typicality). We will state in the abstract and introduction that typicality is defined operationally as local density in the encoder’s space. The introduction already says “an input is typical if its embedding lies in a region the encoder populates densely.” We will add that we compared several candidate properties and kept angular density based on the analyses in Section 4. We will also drop “argue”.

W2, Q2 (PQC degeneracies). PQC is hand-crafted, and it does have these problems. A property that nearly matches the quality has Information near 1, but its Monotonicity is lowered by small fluctuations among saturated bins and is bounded below only by 0.1, so PQC(q, q) falls well short of 1. We have not tested sensitivity to the number of bins, and we will say so. Two points of scope. Within one model, all properties are scored against the same quality q, so their ranking does not depend on how good the model is. Absolute PQC values across models should be compared cautiously, and we will say so in the text. Spearman and AUROC are per-sample and not sign-agnostic, so they do not rank all properties identically to PQC (Appendix, "Alternative Scorings"). We will add a subsection on these degeneracies, including the PQC of the oracle property as a ceiling. We will describe your histogram-distance proposal as a better-defined alternative that we have not evaluated. We do not know whether it would change the ranking, and we will not claim that it would not.

Minor. We will check the Figure 5 label (overlap vs. accuracy) and fix the sentence about Wang & Isola (2020) and Liu et al. (2018).

Q1 (Euclidean density). We have not tested this, and thank you for raising it. For unit-norm vectors the squared Euclidean distance is 2 - 2cos, so the mean squared distance to the same k neighbours is a monotone transform of our score. Without normalization, the Euclidean version mixes angle and norm. Sun et al. likewise find that feature normalization is critical for their score, which is consistent with our angle-only choice. We will state the equivalence and note that the unnormalized variant, the k-th-NN distance and the averaged-k distance are untested. Norm does carry some information, and we will present the angle-only score as one choice among several.

Q3 (instruct vs. base). We have no tested explanation. Instruct PQC is lower (mean 0.18 vs. 0.28). Our setup contains one confound: base models get 5 in-context examples and instruct models get none (Appendix, “LLM Quality Measure”), so the last-token activations come from different prompt formats. We will state this confound in the text. One possible account is that post-training reshapes late-layer geometry. We have not tested it, and we will label it speculation.

The authors have successfully addressed my concerns.

  • The novelty concern, which seems to be shared with reviewer xQSm. Better acknowledgement of existing work and positioning within the literature addresses address it.
  • The rewording of the definition of typicality addresses the slight clarity issue.
  • Additional discussion on PQC: I do not expect the authors to test every possible metric. I maintain that more grounded measures would significantly improve this paper, as also raised by reviewer xQSm. However, this choice, even if not perfect, is well motivated and provides interesting results. The proposed added discussion is sufficient.

While additional analysis (e.g., on the mechanisms driving angular density/clustering) would be a great addition to this work, it can remain as future work.

For this reason, I believe the paper makes a valuable contribution to the literature and should be accepted. I will raise my score accordingly.

Thank you for your follow-up and for raising your score. We are glad that our answers addressed your concerns!

Reviewer_4wLqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether a model’s embedding geometry can predict how reliably it handles an individual input, without requiring labels. The authors propose angular density, which measures the average cosine similarity between an input’s embedding and its nearest neighboring embeddings. They hypothesize that inputs in denser local regions are more typical for the model and are therefore more likely to be handled correctly. They evaluate seven self-supervised vision models, five vision-language models, and nine large language models, measure classification accuracy, cross-model alignment, and performance on five language benchmarks, comparing angular density with embedding norm and other geometric baselines.

Authors find 1) that a higher angular density generally correlates with better performance across the tested models and modalities; 2) that angular density is more consistent than embedding norm in several settings; 3) that the local angular density performs better than the tested global-centroid measure of typicality; and 4) that the text embedding density can also predict the classification reliability of corresponding images.

My biggest critique is that the results are primarily correlational. The claim that angular density measures typicality is plausible but not conclusively established. More independent datasets, uncertainty estimates, comparisons with existing confidence measures, and real-world deployment tests would strengthen the work. Overall, this is a promising empirical study showing that local embedding geometry can provide a useful, label-free signal of model reliability. However, it does not yet establish a general theory of typicality or prove that angular density is a reliable substitute for confidence estimation in practical deployments.

Critical strengths

Strengths:

  1. The paper studies a useful problem: whether embedding geometry can provide a label-free signal of input-level reliability.

  2. Angular density is simple to compute and interpret, using local neighborhoods rather than only distance to a global centroid.

  3. The evaluation spans seven self-supervised vision models, five vision-language models, and nine language models, providing evidence across different model families.

  4. The paper evaluates several outcomes, including k-NN classification accuracy, cross-model neighborhood alignment, and multiple-choice language-model performance.

  5. Comparing angular density with embedding norm and global geometric baselines helps assess whether local angular structure adds information.

  6. The experiments include ablations and robustness checks involving reference-set size, out-of-distribution contamination, class imbalance, and normalization.

  7. The analysis considers both final representations and changes during training, adding evidence beyond a single static evaluation.

  8. The paper reports cases where angular density is more consistent than embedding norm, particularly in the vision-language experiments.

  9. The results support a plausible link between local embedding density and how typical an input is for a model.

10.The method is relatively lightweight conceptually and could be useful where labels or model-provided confidence scores are unavailable

Critical weaknesses

Weaknesses:

  1. The main evidence is correlational; it does not establish that angular density causes or reliably explains correctness.

  2. The experiments cover a limited range of settings, with vision evaluation concentrated on ImageNet-1k and language evaluation focused on multiple-choice benchmarks.

  3. Angular density may reflect familiarity, class frequency, semantic redundancy, or dataset composition rather than typicality alone.

  4. The paper does not establish that angular density is better than common confidence measures such as predictive entropy, logit margin, or likelihood.

  5. Results vary across models, and embedding norm performs better in some settings, limiting the generality of the main claim.

  6. The method depends on a suitable reference set; its reliability may degrade when the reference data are scarce or poorly matched to the evaluation distribution.

  7. The language-model evaluation relies on prompt activations and next-token likelihoods for multiple-choice answers, which may not generalize to open-ended generation.

  8. The interpretation of angular density as typicality remains plausible but is not independently validated against direct measures of input typicality.

Questions for authors

Questions for authors:

  1. What evidence distinguishes typicality from familiarity, class frequency, semantic redundancy, and other factors that could produce high local density?

  2. How does angular density compare with predictive entropy, logit margin, likelihood, and other established confidence or uncertainty measures under the same evaluation protocol?

  3. Can the authors report uncertainty estimates and statistical tests for the main comparisons, including the differences between angular density and embedding norm?

  4. How sensitive are the results to the choice of k, the reference-set size, and the distribution of reference examples?

  5. Do the findings hold on additional vision datasets and distribution shifts beyond ImageNet-1k?

  6. Do the language-model results generalize to open-ended generation and non-multiple-choice tasks, where correctness is harder to measure?

  7. Can the authors evaluate whether angular density improves selective prediction or risk–coverage trade-offs, rather than only correlating with aggregate performance?

  8. How should PQC be interpreted relative to standard rank correlations and AUROC, and how sensitive is it to non-monotonic trends or ties?

  9. Why does angular density outperform norm in some settings but not others, and can the authors identify the conditions that explain these differences?

  10. What evidence supports interpreting angular density as a measure of typicality rather than simply a proxy for local clustering in the chosen embedding space?

1 comment

Dear reviewer,

Thank you for your time and the detailed list.

Correlational evidence; typicality vs. familiarity, class frequency, redundancy, local clustering (W1, W3, W8, Q1, Q10). We show prediction, not causation. We will remove the causal-sounding language and say this in the abstract. “Typicality” is our model-relative, operational reading of local density. Our evidence on this point is partial: the class-imbalance ablation with majority and minority queries (Appendix, “Reference-Set Composition”), neighbour purity vs. density (Appendix), and the global-centroid baseline (Figure 4). None of these separates familiarity or redundancy from typicality, and we use no direct measure of typicality. We will list this as a limitation and describe angular density as a proxy for local clustering in the encoder’s space.

Comparison to confidence measures; selective prediction (W4, Q2, Q7). We have not run this comparison. It is the natural one for a reliability signal, and we will not claim that density is better than entropy, margin or likelihood. Density is useful where those do not exist, since it needs no classifier head, no logits and no labels. We will state that it has not been compared to them and that risk-coverage behaviour is untested.

Uncertainty and tests (Q3). We report no significance tests. The ± in the tables is across models, not seeds or samples, and retraining SSL models with several seeds is not feasible in the rebuttal period. We will state that each SSL method is a single run, and we will add bootstrap intervals over evaluation samples for the main comparisons, including density vs. norm.

Sensitivity to k and reference set (Q4, W6). We ablate k in {5, 10, 20} and the curves are near-identical (Appendix, “Different values of k” and “Cross-Model Alignment”). We also vary reference-set class imbalance and OOD contamination up to 50% (Appendix, “Reference-Set Composition”). Reliability degrades when the reference set is scarce, as the Limitations say. We do not test a reference set from a different domain than the queries, and we will add this to Limitations.

Other datasets (W2, Q5); open-ended generation (W7, Q6). Our vision evidence is ImageNet-1k only, and we will scope the claims accordingly. LLM evaluation uses multiple-choice benchmarks and the next-token likelihood of the answer letters, so our results do not cover open-ended generation. We will say so in Limitations.

Norm vs. density; conditions (W5, Q9). Norm has the higher accuracy-PQC for MAE (0.14 vs. 0.06), LeJEPA (0.38 vs. 0.33) and SimSiam (0.30 vs. 0.29), and for Gemma3 4B, Mistral and Qwen3 1.7B (Table 1). The abstract says norm is “less consistent”, and we will list these exceptions in the text. We do not have a tested account of when each wins, because norm’s direction depends on the training method (Figure 3 and the linear-probe figure). We will present this as an open question.

PQC vs. rank correlation and AUROC (Q8). Spearman and AUROC are in the Appendix (“Alternative Scorings”) and are computed per sample. Density has the highest AUROC and a positive Spearman rho, but Mahalanobis is negative on both for most models, while PQC, being sign-agnostic, scores it highly (e.g. 0.89 on SimCLR). The rankings therefore agree only after sign flips. PQC adds the size of the quality gap between bins.

Reviewer_xQSmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes angular density, the mean cosine similarity of an embedding to its 10 nearest neighbours, as a label-free way to see if an input is well represented. It also introduces the PQC index to score how well a per-input property sorts inputs by quality. The authors test 7 self-supervised vision models trained on ImageNet-1k, 5 VLMs on ImageNet-Captions, and 9 LLMs on five QA benchmarks. They report that inputs with high angular density are more often classified or answered correctly and, for vision and VLMs, have more similar neighbourhoods across models.

Critical strengths

The study is broad in models and paradigms. It covers three paradigms and 21 models, and for the SSL models the authors train from scratch and follow the relation over training checkpoints.

For cross-model alignment the result is consistent. Angular density has the highest PQC of the four properties for all 7 SSL models (shown in Table 6).

For classification accuracy, angular density is, on average, as good as the per-class Mahalanobis distance, which needs labels, while it uses no labels (shown in Table 6).

Critical weaknesses

Please check the questions.

Questions for authors

clarity of the writing: The writing of the paper is not very clear. Especially the abstract and introduction are hard to read and do not make the contribution and the goal very clear. Many terms are used without definition: "property", h in L236, f vs f^(t), and PRH. Figure 2 needs an explanation of each part. The paper would benefit from rewriting. There is a missing citation in line 1244. Also, line 1243 is broken.

Definition 3.1 and PQC index: The PQC index is the main measuring tool, but it has no citation or derivation, and it is not validated as a measure. The only motivation is two informal requirements (L178–186). It also raises a question: how would a random baseline perform? Monotonicity is at least 0.1 by construction, because the first bin always counts. It is also hard to judge how a value of this index really is and what it means. Defining a new metric would benefit from deeper analysis and theoretical support.

Comparison: For classification accuracy, angular density is about as good as the norm and the label-based Mahalanobis distance (Table 6: 0.40 ± 0.18 vs 0.37 ± 0.13 vs 0.39 ± 0.31).

One dataset: All vision and vision-language experiments use only ImageNet, and the SSL models are evaluated on the data they were trained on. Using only one dataset is not enough for a claim about a general, label-free signal.

measure vs model: The strongest result may partly come from how the measures are built. Angular density, the k-NN classifier and the k-NN overlap all rely on 10-nearest-neighbour structure on the same normalised embeddings, as the authors say themselves (L231–233). An embedding in a dense region has close and stable neighbours, so another model is likely to find the same neighbours, and they are likely to share its class. The linear probe in D.1 checks this only for accuracy, and only as a figure without numbers. For cross-model alignment, where angular density wins most clearly, there is no check that does not use neighbours.

Novelty: I am concerned about the novelty of the paper. The paper says no prior work gives a diagnostic that is per-input, label-free, cross-modal, and local (L64–69). But per-input, label-free, and local density scores already exist. TypiClust (Hacohen et al., ICML 2022) defines the typicality of an example as the inverse of its mean distance to its nearest neighbours in self-supervised embeddings, which is a similar idea, and it is not cited. Sun et al. (2022) score inputs by the k-NN distance on normalised features, and this score uses no class labels.

VLM results: The VLM results are not surprising and not clearly described. CLIP, OpenCLIP and SigLIP are trained to align images and captions, so text density predicting image accuracy is expected. The paper does not show that text density adds anything beyond image density. LLaVA uses a CLIP ViT-L/14 image encoder, so it is not independent of CLIP in the alignment measure. It is also not clear what the norm means for these models, since they normalise the embeddings in the loss, and the paper does not say which embedding is used.

3 comments

Dear Reviewer,

Thank you for your time and careful feedback, that allows us to improve our paper.

Clarity. We will rewrite the abstract and introduction around one definition: an input is typical if its embedding lies in a densely populated region of the encoder’s space. “Property,” the classifier h, and f vs. f^(t) will be defined at first use, and we will expand the Figure 2 caption part by part. We will also fix the broken line and the missing citation in the appendix.

PQC (Def. 3.1). Your concern about PQC is reasonable. PQC is a constructed index built to meet two requirements. On the random baseline: PQC is Information times Monotonicity, so it cannot exceed Information. A property unrelated to quality puts similar accuracy in the top and bottom bins, which drives Information, and therefore PQC, toward zero even though Monotonicity is at least 0.1. Most reported PQCs (0.2 to 0.6) are far from that, but not all: MAE density is 0.06, VLM norm is 0.00 to 0.11, and invariance is 0.00 to 0.19. We have not run a permutation test and will add one to the revision, so that the low values can be read against a null. Spearman's rho and AUROC (Appendix, "Alternative Scorings") are computed per sample, and PQC is sign-agnostic (it uses the absolute bin gap), so the three do not give identical rankings: for example, per-class Mahalanobis has negative per-sample rho and AUROC below 0.5 for most models, yet a high PQC. We will state this and add a paragraph on how to read PQC values.

Comparison. You are right that on SSL accuracy the three scores are close (0.40 / 0.37 / 0.39). The differences lie elsewhere. For cross-model alignment, density scores 0.30, against 0.09 for norm and 0.12 for Mahalanobis (Table 6). For VLM text embeddings it scores 0.23, against 0.05. On accuracy, Mahalanobis beats density on SimSiam, SimCLR and MoCov3, and its ± is large (0.31), so the tie rests on a few models; we will say so rather than state that density "matches or exceeds" it on most models. Norm also reverses direction during training for DINOv2, DINOv3 and MoCov3, and density needs no labels. We will reword the claim to “matches on accuracy, label-free, more stable across settings.” The reported ± is across models, not samples or seeds. We will say so and add confidence intervals.

One dataset. We will scope our claims to ImageNet-1k and list this under Limitations. On the second half of your point: the k-NN classifier is fit on ImageNet train embeddings and evaluated on validation images, which the SSL models never saw. Angular density and norm are computed within the validation set. A train-to-train ablation of the classifier protocol gives near-identical curves (Appendix, "Training vs validation"). It does not test where density is computed, and we will state this.

Measure vs. model. This concern holds partly for alignment, and we will address it in three parts. (i) For LLMs, quality is benchmark correctness from answer-token log-likelihood. That uses no neighbour structure, and density still predicts it (Table 1). (ii) The linear-probe check (Appendix D.1) is monotone for all 7 models. It currently appears only as a figure, so we will add the numeric PQC. (iii) For cross-model alignment we have no neighbour-free check. We will say so plainly as a limitation: that result shares structure with the score.

Novelty. You are right about the prior work. TypiClust and Sun et al. are close label-free density scores, and we wrongly described Sun et al. as label-based. We will cite TypiClust, correct our description of Sun et al., and narrow our claim to what we believe is new: a per-input evaluation of one geometric score across vision, vision-language and language models, the local-vs-global contrast (Figure 4), and the text-to-image result. We do not claim a new density estimator.

VLMs. Text density predicting image accuracy is less surprising for contrastively aligned models, and we will say so. We have not tested whether text density adds information beyond image density, and we will state that as open. LLaVA’s image encoder is CLIP ViT-L/14 (Appendix, VLM details), so it is not independent of CLIP. We will note this and exclude LLaVA from independence arguments. We will also state which embedding (layer, pooling, pre- or post-projection) each VLM uses for norm and density.

Thanks to the authors for providing the rebuttal.

While the rebuttal answers some questions, I am still unsure about some. My concern about PQC remains unresolved. As the paper defines a measurement protocol that is the multiplication of two metrics, it would be beneficial to analyze it more. The questions I asked are still unanswered. Please correct me if I am wrong. The missing related work and novelty also make the novelty of the paper narrower. In addition, not sure how trustworthy reporting results only on ImageNet is.

Dear Reviewer,

Thank you for your follow-up. We have now added additional analysis to address your questions.

Please find a summary of our points below:

  • PQC is above a permutation null in all 14 cells, and its ranking is stable across aggregations and bin counts. The main point of it being multiplicative is that one requires both monotonicity and a high range of values in order to have high PQC scores.
  • We also note that PQC is more consistent than other measures: higher density means higher accuracy in all 7 models, while the embedding norm reverses sign in 3 models.
  • All numbers are for the 7 SSL models on ImageNet-1k validation (50k images), final checkpoints, with classification accuracy as the target, comparing norm and angular density.

We go into more details on each of these points below:

Definition 3.1 and PQC index

We read your questions as: (1) is PQC derived or validated, (2) how does a random baseline perform, (3) what does a value mean. Please tell us if we misread any.

On (1): PQC is hand-crafted. It has no derivation, and we do not claim one, so we validated it empirically. On (2): Monotonicity is at least 0.1, as you note, but a random property has Information near zero, so the product stays near zero. Over 2000 shuffles per (model, property) cell, the null PQC mean is 0.002 to 0.003 and the 95th percentile is at most 0.009. Observed PQC ranges from 0.063 (MAE, angular density) to 0.610 (DINOv2, angular density), and every cell exceeds all 2000 null draws (p < 0.001). On (3): Information is the absolute accuracy gap between the lowest and highest property bins, and PQC discounts it by monotonicity. When Monotonicity is 1, a PQC of 0.5 means the extreme bins differ by 50 accuracy points, in either direction. Values below about 0.01 are indistinguishable from chance. Monotonicity is lowered by fluctuations among saturated bins, so this oracle falls short of 1. Within one model, all properties are scored against the same quality, so their ranking does not depend on how good the model is; absolute values across models should be compared with care, and we will say so.

Regarding the product between the two variables in the PQC calculation, Monotonicity is 1.0 in 11 of 14 cells, so PQC is almost exactly Information there. Using the sum, min or harmonic mean, or 5 or 20 bins, changes the angular-vs-norm ordering only for SimSiam, where the difference is not significant. We had not tested bin sensitivity at the time of our first responses; these results now cover it.

Regarding other scorings

Rank correlation with 10-bin PQC is 0.97 to 1.00 across bin counts and aggregations, but 0.57 against per-sample Spearman and AUROC. The cause is sign. PQC is sign-agnostic, and for MoCov3, DINOv2, and DINOv3, norm has negative Spearman (−0.27, −0.37, −0.23) and AUROC below 0.5 while scoring PQC 0.31 to 0.50. Angular density has positive Spearman and AUROC above 0.5 in all 7 models, so its direction can be read the same way everywhere. We already report Spearman and AUROC in the Appendix.

Confidence intervals

We bootstrapped over samples (1000 resamples, 95%); this covers evaluation variance only, not training seeds. Angular density minus norm in PQC is positive for SimCLR (+0.018 [CI]), DINOv2 (+0.107 [CI]) and DINOv3 (+0.179 [CI]), negative for MAE (−0.078 [CI]) and LeJEPA (−0.044 [CI]), and not significant for SimSiam and MoCov3. This confirms the wording from our first response: on accuracy, angular density matches norm rather than beating it. Its advantages are a consistent sign, no labels, and cross-model alignment. We will revise the abstract and introduction accordingly.

Novelty

We appreciate the suggested references and will include them. TypiClust (Hacohen et al., 2022) defines typicality as the inverse of the mean Euclidean distance to an example's nearest neighbours in self-supervised embeddings, and uses it to choose which examples to label in low-budget active learning. Sun et al. (2022) score a test input by the distance to its k-th nearest neighbour among L2-normalized features, for OOD detection; the score uses no labels at test time, though their encoder is trained with labels. Ours is the mean cosine similarity to k neighbours on mean-centered embeddings. On normalized embeddings, Euclidean distance is a monotone function of cosine similarity, so these scores are close, and we will say so. We have not compared them directly, and we will state this. We will cite TypiClust, correct our description of Sun et al., and claim no new typicality score. What we believe remains is the use: predicting per-input quality, rather than selecting samples or detecting OOD inputs.

We will scope all vision and VLM claims to ImageNet-1k.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.