acceptodds
Under review as a conference paper at ICLR 2027

Vision Transformers Learn Word Meaning from Images Alone

Abstract

Can a vision model learn what words mean from pixels alone? We show that the two largest DINOv3 vision transformers organize words rendered in images by word meaning, despite being pretrained using self-supervision on images only, without paired captions, class labels, or other language supervision. We derive word representations from embedding differences between word images and matched controls, then test whether semantically related words are neighbors within each model's embedding space, while controlling for spelling similarity. We find that brand names in the same industry group together, as do nouns from the same semantic domain. Country-capital pairs exhibit consistent vector offsets, similarly to word embeddings trained on text. These word representations also align with the representations of the visual object categories they name. DINOv3-H and DINOv3-7B are, to our knowledge, the first self-supervised vision models to show substantial semantic structure for rendered words. Comparisons with text and speech encoders reveal shared structure among representations of the same words. Contrastive alignment with either encoder further strengthened DINOv3-H’s word-level semantic structure, while leaving the other finetuned vision models largely unchanged. Image-only training can therefore teach vision models not just what words look like, but also how their meanings relate.

Primary area:
interpretability and explainable AI
Keywords:
vision transformers, self-supervised learning, interpretability, semantic representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7ta7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether vision models pretrained through image-only self-supervision can learn semantic relationships among rendered words. The authors construct word directions from the difference between representations of word images and matched controls, and evaluate category structure, country–capital relations, synonym relations, and word–visual-class alignment. The results show clear word-level semantic structure in DINOv3-H and DINOv3-7B, whereas DINOv2, MAE, and Web-DINO largely remain close to an untrained ViT baseline.

Critical strengths
  1. The question studied and the phenomenon discovered in this paper are very interesting. The absence of explicit language supervision does not necessarily prevent a model from acquiring language-related knowledge from scene text and visual context. Understanding such unexpected capabilities that emerge with scale is important for both self-supervised learning and the evaluation of multimodal systems.
  2. The word–class alignment results in Table 2 are convincing. DINOv3-H achieves word-to-class and class-to-word top-10 accuracies of 0.645 and 0.68, respectively, compared with only 0.005 and 0.045 for the untrained ViT-B. Appendix D.3 further shows that the matched cosine similarity remains almost unchanged when ImageNet images with detected text are excluded (0.234 to 0.23).
  3. The core method is clear. By subtracting a matched phase-scrambled control, the authors control reasonably well for background and rendering effects. The untrained ViT baseline, spelling clusters, and word-length-conditioned permutation tests also reduce the likelihood that the main results can be explained solely by visual similarity between word forms.
Critical weaknesses
  1. The evidence for “consistent vector offsets” is insufficient. Figure 1b shows only six hand-selected examples from the 60 country–capital pairs. Appendix C.1 further explains that the horizontal axis is defined using the mean offset of the undisplayed pairs and is visually stretched by a factor of 1.6. Meanwhile, Section 7 acknowledges that these offsets do not improve analogy retrieval. I understand that Figure 1b may primarily be intended as an illustration. Nevertheless, the current evidence is insufficient to support the claim in the Abstract and Section 1 that the model exhibits vector-offset relations comparable to those in classical word embeddings. The authors should quantify offset consistency over all pairs and compare it with random-pair or label-shuffled baselines. Otherwise, this claim should be weakened.
  2. “Learn word meaning” is broader than what the experiments directly establish. In Table 1, synonym top-10 accuracy is only 0.06 for DINOv3-H and 0.1 for DINOv3-7B. Appendix C.1 and Table 5 also show that, after correction, no vision model reliably ranks synonyms above antonyms. The current results provide strong evidence for categorical and factual associations and for word–visual alignment. However, they may still arise from word identification combined with scene-text co-occurrence in the training images, a possibility also acknowledged in Section 7. The Abstract, and Conclusion should therefore distinguish more clearly among lexical association, visual grounding, and broader word meaning.
Questions for authors
  1. Can the authors quantify offset consistency over all 60 country–capital pairs? If these offsets do not improve analogy retrieval, what do they establish beyond ordinary country–capital association?

  2. Were the vocabularies, relation pairs, and class names described in Appendices A and D.1 fully fixed before inspecting the results of any evaluated model?

1 comment

Thank you for the careful review, 7ta7, and the kind words. We're glad you found the core method clear and the word-class alignment results convincing.

W1. Evidence for consistent vector offsets

We quantified offset consistency over all 60 country-capital pairs. For each pair, we take the cosine between its country-to-capital offset and the mean offset of held-out pairs (five folds), and average over pairs. As you suggested, we compare this with 999 random re-pairings of countries and capitals.

ModelOffset cosineRe-pairedp
Untrained ViT-B0.0120.0090.400
DINOv3-H0.1210.0430.001
DINOv3-7B0.1330.0610.001
CLIP-B0.2450.1710.001
Qwen3-8B (text)0.4890.3280.001

The offsets are clearly more consistent than under random re-pairing in both DINOv3 models, CLIP-B and Qwen3-8B, but not in the untrained ViT-B. The six pairs in Figure 1b therefore illustrate a regularity present across all pairs.

We will keep the offset finding, but the Abstract and Introduction will no longer compare the offsets to those of word embeddings trained on text. The full analysis will be added to Appendix C.

W2. "Learn word meaning" is broader than the experiments

The paper now makes this distinction explicit (see our response to Z4e4, W2). It defines word meaning as lexical meaning and tests two parts: how a word relates to other words, and whether it relates to images of the category it names. The Abstract now names these two parts more generally for readability, and the Conclusion details the distinction in our claims following your recommendations.

Synonymy is the weakest of the relations we test, as you note. Section 7 now gives an account of why: the models may learn which words occur together, and synonyms tend to replace each other rather than appear together. In line with this, DINOv3-H relates some synonyms of class names to the right photographs, such as "maize" for the class corn (see our response to e2jz, W1).

We also tested the role of text in the training images directly (see our response to reviewer Z4e4, W3). Of two Web-DINO-3B models trained with the same objective, the one trained only on images that contain text retrieves a class from its written name (word-to-class top-10 of 0.355), while the one trained on all images does not (0.010, against 0.005 for the untrained ViT-B). Text in the training images is therefore one source of the structure, as you suggest.

Our Spanish experiment (see our response to reviewer e2jz, W2) shows that the grounding does not depend on English spelling: in DINOv3-H, Spanish class names that share no letter pairs with their English names still retrieve their class (20 of 82 in the top ten, against 0.8 by chance), whereas category structure among such Spanish nouns is not detectable.

Q1. Offset consistency over all 60 country-capital pairs

Please see response in W1. We read the offsets as a geometric regularity of the country-capital association, not as analogical reasoning. This agrees with Section 7, which states that the offsets do not improve analogy retrieval.

Q2. Were the word lists fixed before any result was inspected?

Yes. The vocabulary and the relation pairs were fixed before any model was evaluated, and the class names were fixed by the rules in Appendix D.1 before any model was evaluated on them. None of these rules uses the output of an evaluated model. Two later choices are disclosed in the paper: the six pairs shown in Figure 1b and six two-clue pilot grids, which are excluded from the results. The lists added for this rebuttal were likewise fixed before rendering.

For the discussion period, we have updated the main text of the PDF (in blue), and we will add the new analyses to the appendices in full for the camera-ready version. We hope these changes address your questions and that you will consider raising your score.

Reviewer_Z4e4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates an interesting question of whether vision transformers learn word meanings from images alone, without explicit language supervision such as text captions or labels.

For this, they experiment with 14 models, primarily self-supervised vision transformers such as DINOv3, DINOv2, and MAE, and secondarily also include language-supervised models like CLIP.

The main methodology is obtaining word embeddings from images: rendering words onto different backgrounds, subtracting embeddings of corresponding control images with scrambled lettering, and averaging and centering the resulting representations to isolate the contribution of the written word from background and rendering effects.

Then there are several analyses: checking word-word relationships, word-image alignment, comparisons with text and speech encoders, fine-tuning with language supervision, and so on. DINOv3 models seem to stand out among the self-supervised models. Particularly interesting is that country-capital associations and word-image alignment are strong, whereas synonym retrieval is relatively weak. Overall, the findings seem to suggest that vision models can learn certain semantic relationships without explicit language supervision, with the chosen experiments hinting the boundary of general language understanding.

Critical strengths
  1. Quite interesting question, especially from an abstraction perspective. Surprising that models trained on images alone can develop semantic relationships between words without explicit language supervision.
  2. Thoughtful and diverse set of experiments. The paper investigates the question from multiple angles, including word–word relationships, word–image alignment, cross-modal comparisons, and fine-tuning.
Critical weaknesses
  1. Difficult to follow. I found the paper quite difficult to follow: way too many experiments, without a clear hierarchy of what to expect. Which of these experiments is central to the claim, and which are supporting experiments? In sum, while the experiments are rigorous and seem to follow a story in how the authors approached the problem, the synthesis is poor, and that story does not come out easily. The paper demonstrates several interesting phenomena, but doesn't fully develop a unified account of what those phenomena come together to tell us generally about semantic representations in vision models.

  2. Unclear definition of "word meaning" given that it is in the literal title. What exactly is "word meaning" in this context? While the paper operationalizes this through several experiments, the notion seems to shift throughout the paper. This is especially problematic since the title makes a statement rather than asking a question: a statement without a precise definition is difficult to evaluate. Further, each finding seems to tell a different story: country-capital associations and word-image alignment seem strong, synonym not so much. It is not clear how we should interpret these different results together, or to what extent they collectively justify the broader claim of learning "word meaning".

  3. Limited explanation of why some models do so well. The findings immediately ask a question: why and how do certain models perform so well in these experiments? Leaving that open feels unsatisfactory. Is it the training objective, the data (e.g., exposure to similar text images) or something else?

Questions for authors

None

2 comments

Thank you for the careful review, Z4e4, and the kind words. We share your view that the question is interesting, and we're glad you appreciated the experimental design. Clarity matters a lot to us, and we've made several changes to make the paper easier to follow. If particular parts were confusing, we'd love to hear about them so we can fix those too during the discussion period.

W1. Too many experiments without a clear hierarchy

The last paragraph of the Introduction now states the hierarchy. We report three results, each in one section with a matching appendix. The first is our main result, and the other two build on it: (1) lexical structure, semantically related words are neighbors when spelling is controlled (Section 4, Appendix C); (2) word-class alignment, rendered words align with the object categories they name (Section 5, Appendix D); (3) representational alignment with text and speech encoders, and finetuning (Section 6, Appendix E). Appendices A and B describe the data and measures for all three.

W2. Definition of "word meaning"

By word meaning, we mean lexical meaning (Murphy, 2010). The Introduction now says so, and a new paragraph at the start of Section 3 defines the two parts we test: how a word relates to other words, through similarity and association (Hill et al., 2015), and grounding, whether a written word relates to images of the category it names (Harnad, 1990). With this definition, we hope the title now has a more precise sense, and we are open to narrowing it if you still find it too broad. On a related note, following Reviewer 7ta7's feedback, we have also altered the Abstract and Conclusion to be more narrow in our claims.

Section 7 now gives one account. The models may learn which words occur together, with each other and with the objects they name. Countries appear with their capitals and members of a category in similar scenes, whereas synonyms tend to replace each other rather than appear together. If this is right, synonyms should be easier to retrieve for concrete words than for abstract ones, because two concrete synonyms appear next to the same objects even when they do not appear together.

W3. Why some models do so well

Our main hypothesis is the training data, as Section 7 discusses. DINOv3's data is drawn mainly from about 17 billion images from public Instagram posts, of which 1.7 billion were selected by balanced sampling across clusters of images (Siméoni et al., 2026). We expect many of these images to contain text, which would expose the model to words that occur together and to words next to the objects they name. The DINOv3 authors also report that DINOv3 clearly outperforms DINOv2 on classification that requires character recognition, such as traffic signs and logos. Their training data is not public, so we cannot test this directly.

We therefore tested the role of the data in the closest setting available, with further Web-DINO checkpoints (Fan et al., 2025). They share the DINOv2 objective and a pool of two billion web images, and the 2B and 3B models were trained for the same number of samples, either on all of the images or only on the half judged to contain readable text.

ModelTraining imagesNoun purityCapital top-10Word-to-class top-10Hypernym purity
Untrained ViT-Bnone0.1830.0170.0050.026
Web-DINO-2Ball0.2000.0330.0300.030
Web-DINO-2Bwith text0.2370.0670.1550.435
Web-DINO-3Ball0.2040.0330.0100.057
Web-DINO-3Bwith text0.2830.0500.3550.487
Web-DINO-7Ball, 4x longer0.3010.3000.3900.422
DINOv3-HLVD-1689M0.2690.6000.6450.478
DINOv3-7BLVD-1689M0.3080.6000.3250.209
Chance0.1640.0060.0100.035

Bold marks values more than two standard errors above the untrained ViT-B. Noun purity is the share of a noun's ten nearest words in its category. Hypernym purity is the share of the ten classes nearest a word such as "reptile" that fall under it (see our response to e2jz, W1).

At 2B and 3B, the models trained on all of the images are not detectably above the untrained ViT-B, while those trained on images with text group nouns, relate hypernyms to their classes and retrieve a class from its written name, reaching DINOv3-H's noun and hypernym purity at 3B. With the objective and the amount of training held fixed, this points to the data (the models trained on images with text see each image about twice). Data is not the only factor: Web-DINO-7B, a larger model trained four times longer on all of the images, also shows the structure and is the only Web-DINO model that retrieves capitals.

For the discussion period, we have updated the main text of the PDF (in blue), and we will add the new analyses to the appendices in full for the camera-ready version. We hope these changes address your questions and that you will consider raising your score.

Thank you again for your review. We have posted a brief summary of the revision above, and we would be glad to answer any remaining questions before the discussion period ends.

Reviewer_e2jzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether vision transformers trained on images alone can learn the meaning of words without explicit language supervision. The authors extract word representations by comparing images with rendered words and matched control images, then averaging and centering the embedding differences. Experiments show that DINOv3-H and DINOv3-7B capture some semantic relationships, including noun and brand categories, country–capital associations, and limited synonym relationships. The paper also shows that word representations can align with their corresponding visual object categories. Comparisons with text and speech encoders reveal some shared representational structure, while contrastive finetuning improves synonym retrieval and noun-category grouping in DINOv3-H.

Critical strengths
  1. Novel evidence of lexical semantic structure emerging from image-only pretraining. The paper shows that DINOv3-H and DINOv3-7B capture semantic relationships among rendered words without explicit language supervision. In particular, both models achieve 60% top-10 country–capital retrieval accuracy and show meaningful noun and brand category grouping, substantially outperforming earlier self-supervised vision models. The spelling controls and comparisons with untrained models further strengthen these findings. These results reveal interesting and previously under-explored capabilities of image-only pretrained models.
  2. Strong alignment between rendered words and visual object categories without explicit alignment training. The paper demonstrates that representations of rendered words align with the visual categories they name, without requiring language supervision or a learned mapping between word and image representations. DINOv3-H achieves 64.5% word-to-class top-10 retrieval accuracy, compared with 1% chance performance, and its matched cosine similarity is comparable to CLIP-B. This provides compelling evidence that the learned word representations are connected to visual concepts rather than merely capturing similarities among written words.
Critical weaknesses
  1. Limited coverage of lexical semantic relationships. The strongest results mainly focus on category membership and country–capital associations, while performance on other semantic relationships remains relatively weak. For example, synonym top-10 accuracy is only 6% for DINOv3-H and 10% for DINOv3-7B. Although the paper also shows strong word–visual-class alignment, it lacks a systematic evaluation of other semantic relationships, such as hierarchical, functional, and attribute-based relations. The additional two-clue analysis also provides limited evidence of combining semantic information. These limitations make it difficult to determine how broadly the observed associations reflect general word-level semantic understanding.
  2. Unclear origin and generalizability of the observed semantic structure. Although the models are pretrained without explicit language supervision, their training images may contain embedded text. The paper does not provide experiments to distinguish whether the observed semantic relationships mainly come from visual context or textual co-occurrence within images. In addition, the evaluation mainly uses selected English words under controlled rendering conditions, without testing natural scene text or other languages. These limitations make it difficult to assess the extent to which the findings generalize beyond the current experimental setting and what drives the emergence of the observed semantic structure.
Questions for authors
  1. How sensitive are the main results to the vocabulary used for centering? Would changing the vocabulary size or composition significantly affect the results?
  2. Would the alignment between DINOv3 and text encoders remain significant after controlling for character-level spelling similarity, rather than only word length?
3 comments

Thank you for the careful review, e2jz, and the kind words. We're especially glad you found the paper clear and informative, and that you appreciated the novelty of the findings, the spelling controls, and the strong word–visual-class alignment. Thank you also for the thoughtful questions, which we've done our best to address below. In particular, we find that changing the centering vocabulary's size and composition has a measurable and uniform effect accross models, slightly lowering performance, but leaving the main conclusions unchanged. We also address your question about controlling for spelling similarity in cross-encoder alignment.

W1. Coverage of lexical semantic relations

We agree that the submission covered a limited set of relations. To address your concern, we have now tested hierarchical relations with 23 hypernyms, such as "reptile" and "furniture", that each cover several ImageNet classes and name none of them. Hypernym purity is the share of the ten classes nearest a hypernym that fall under it.

ModelHypernym purity
Untrained ViT-B0.026
DINOv3-H0.478
DINOv3-7B0.209
CLIP-B0.709
SigLIP2-B0.652
Chance0.035

DINOv3-H relates a hypernym to the classes under it, and DINOv3-7B does so more weakly.

On synonyms, we agree that retrieval among words is weak. We have now also tested whether a synonym of a class name, such as "maize" for the class corn, relates to that class's photographs. DINOv3-H does this for some synonyms, about as often as CLIP-B, while DINOv3-7B does not. Synonymy therefore remains limited, though in DINOv3-H it is partly visible through grounding.

We did not test functional or attribute-based relations, and we agree that the two-clue analysis gives limited evidence of combining clues. The paper now states that by word meaning we mean lexical meaning, and a new paragraph at the start of Section 3 defines the two parts we test: how a word relates to other words, and whether it relates to images of the category it names (see our response to Reviewer Z4e4, W2). We will state there that functional and attribute-based relations are untested. We will add the hypernym and synonym results as an appendix.

Q1. Sensitivity to the centering vocabulary.

The results are slighly sensitive to the centering, but it does not change the main conclusion. In Eq. (1), subtracting the vocabulary mean removes the component shared across word directions, with the aim of making word-specific structure clearer. Our new experiment show that while centering improves several scores, the central findings persist without it and remain robust to changes in the centering vocabulary’s composition and size. We tested this by removing centering entirely.

The table reports centered -> uncentered scores. For word-image comparisons, we remove centering of the class-name word directions while keeping the visual directions fixed.

MeasureDINOv3-HDINOv3-7BCLIP-BSigLIP2-B
Noun neighbor purity0.269 -> 0.2360.308 -> 0.2550.297 -> 0.2850.318 -> 0.293
Brand neighbor purity0.351 -> 0.2690.334 -> 0.2760.343 -> 0.3320.299 -> 0.285
Capital top-10 accuracy0.600 -> 0.5830.600 -> 0.5170.550 -> 0.5830.183 -> 0.167
Synonym top-10 accuracy0.060 -> 0.0400.100 -> 0.0800.200 -> 0.1930.060 -> 0.040
One-edit top-10 accuracy0.607 -> 0.5270.620 -> 0.5400.853 -> 0.8270.947 -> 0.933
Plural-or-base top-10 accuracy0.813 -> 0.7870.853 -> 0.8670.987 -> 1.0001.000 -> 1.000
Matched word-image cosine0.234 -> 0.1740.120 -> 0.1210.237 -> 0.1990.184 -> 0.111
Word-to-class top-10 accuracy0.645 -> 0.5550.325 -> 0.3200.855 -> 0.6900.630 -> 0.355
Class-to-word top-10 accuracy0.680 -> 0.7000.405 -> 0.3850.845 -> 0.8500.735 -> 0.705
Word-to-class median rank (lower is better)1 -> 3154 -> 153.51 -> 22 -> 42.5
Class-to-word median rank (lower is better)1 -> 132.5 -> 321 -> 12 -> 2

Removing centering reduces some scores, but categorical and factual associations and visual grounding remain in DINOv3-H and DINOv3-7B. Sensitivity to centering also occurs in both language-supervised reference models. We have clarified the role of centering in Section 3.3 of the updated preprint and will include the full ablation in the appendix in the final paper.

W2. Origin and generality of the structure

On the origin of the semantic lexical structure, we do not propose any special mechanism. Our hypothesis is simply that the structure comes from training data in which many images contain text, such as the public Instagram posts that DINOv3 is trained on. Our response to Reviewer Z4e4 (W3) tests this. There we compare Web-DINO checkpoints that share the DINOv2 objective and the same pool of two billion web images, trained either on all of the images or only on the half that contain readable text. The models trained on all of the images are not detectably above an untrained ViT-B on any of the four measures reported there. Trained only on images with text, the models group nouns by category and retrieve an ImageNet class from its written name (word-to-class top-10 of 0.355 at 3B, against 0.010 when trained on all images). This supports that text in the training images could be a source of the structure, possibly through words seen next to other words and next to the objects they name. To address your concern about other languages, we have now translated our word lists into Spanish (573 words).

We report only the class names and nouns that share no pair of adjacent letters with their English translation, such as iglesia and church. For brevity, the table shows a subset of the models, and the camera-ready version will report all models in an appendix.

ModelMatched cosine, EnglishMatched cosine, SpanishTranslation top-10
Untrained ViT-B0.0100.0170.028
DINOv3-H0.2210.0720.170
DINOv3-7B0.1270.0220.191
CLIP-B0.2390.0210.014
SigLIP2-B0.1870.0280.000
Qwen3-8B (text)--0.908
Chance0.0000.0000.017

Matched cosine is the cosine between a class name's word direction and its class's visual direction (82 class names). Translation top-10 is the share of 141 Spanish words whose English translation is among their ten nearest English words. Bold marks image-model values more than two standard errors above the untrained ViT-B.

DINOv3-H retrieves the class of 20 of 82 Spanish class names in the top ten, against 0.8 by chance, and this holds on photographs without text. Both DINOv3 models also place Spanish words near their English translations, while CLIP-B and SigLIP2-B do not. Once neighbors that share letter pairs are excluded, no image model groups the Spanish nouns by category, although Qwen3-8B does. We will add these results as an appendix and state that the structure is shown for English and extends only in part to Spanish.

We would also like to test natural scene text, but have so far not found a clean way to do so. We would either have to search a large dataset for photographs in which the words happen to appear, where a word is often confounded with the scene around it, or create the images with generative AI, which brings in the biases of the image generator.

Q2. Alignment with text encoders after controlling for spelling

Yes. We regressed character-level spelling similarity (shared adjacent letter pairs, shared letters, edit distance) and word length out of the similarities of both the vision model and the text encoder, and recomputed the alignment.

ModelOriginalSpelling removed
Untrained ViT-B0.0200.010
DINOv3-H0.1220.081
DINOv3-7B0.1350.084
CLIP-B0.2470.192

Mutual 10-NN alignment with Qwen3-8B (chance 0.006). For brevity, the table shows a subset of the models.

DINOv3-H and DINOv3-7B keep more than 60% of their alignment, about eight times that of the untrained ViT-B, and it remains significant (p = 0.001). The other two Qwen3 encoders give the same result. This control is conservative, since it also removes agreement on words related in both form and meaning, such as road and roads, and it depends on which spelling measures are chosen. We therefore keep the original measure in the main text, and the camera-ready version will report this control for all models in Appendix C.

For the discussion period, we have updated the main text of the PDF (in blue), and we will add the new analyses to the appendices in full for the camera-ready version. We hope these changes address your questions and that you will consider raising your score.

Thank you again for your review. We have posted a brief summary of the revision above, and we would be glad to answer any remaining questions before the discussion period ends.

Comments on the paper

1 comment

Thank you all for your careful reviews. We were glad that you found the central question interesting and the paper informative, and we appreciated your positive comments on the novelty of the findings and the experimental results. Your feedback also helped us see where the paper needed clearer explanations, more precise claims, and additional experiments. We've revised the submission PDF, with changes to the main text marked in blue, and wanted to briefly summarize the main changes alongside our individual responses, where we address every point raised.

Our main finding is that the two largest DINOv3 models, trained on images alone, place written words near other words with related meanings, even after controlling for spelling similarity. These words also align with representations of the objects they name.

  • Clearer presentation and definitions. We've revised the "Introduction" section to make the three main results easier to follow and identify the first as our primary contribution. We now define word meaning as lexical meaning (Murphy, 2010), and Section 3 explains the two aspects we test. 1) relations among written words 2) their grounding in images of the objects they name. We're also open to narrowing the title if it still seems too broad.

  • Where the structure comes from. Several reviewers asked whether the structure comes from the training objective, visual context or text in the training images. We compared Web-DINO models trained with the same objective and number of training samples, using either all web images or only images containing text. For the 3B model, word-to-class top-10 accuracy rises from 1.0% to 35.5% when training only on images with text. The larger Web-DINO-7B also shows this structure when trained on all images, suggesting that both training data and scale matter. We now discuss these findings in Section 7.

  • More precise claims. We've revised the "Abstract" and "Conclusion" secitons to show the difference between lexical associations among words and their grounding in visual objects. We also measured country-capital offset consistency across all 60 pairs and found that it exceeds random pairing. The Abstract no longer compares these offsets with those of text-trained word embeddings. We now discuss the limited synonym results more directly and clarify which semantic relations remain untested.

  • Additional tests and robustness. We have added experiments on hypernyms (from WordNet) and Spanish words. DINOv3-H relates hypernyms to the visual classes they cover, and some word-class alignment carries over to Spanish, although noun-category grouping does not. We also found that the main results hold without centering the word vocabolary and that alignment with text encoders remains after controlling for spelling similarity.

The new analyses will be included in full in the camera-ready appendices. We believe these revisions address the main concerns raised and make our findings and their limitations clearer. We hope to hear from you before the discussion period ends. If anything remains unclear, we'd be happy to clarify it before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.