- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether vision models pretrained through image-only self-supervision can learn semantic relationships among rendered words. The authors construct word directions from the difference between representations of word images and matched controls, and evaluate category structure, country–capital relations, synonym relations, and word–visual-class alignment. The results show clear word-level semantic structure in DINOv3-H and DINOv3-7B, whereas DINOv2, MAE, and Web-DINO largely remain close to an untrained ViT baseline.
Critical strengths
- The question studied and the phenomenon discovered in this paper are very interesting. The absence of explicit language supervision does not necessarily prevent a model from acquiring language-related knowledge from scene text and visual context. Understanding such unexpected capabilities that emerge with scale is important for both self-supervised learning and the evaluation of multimodal systems.
- The word–class alignment results in Table 2 are convincing. DINOv3-H achieves word-to-class and class-to-word top-10 accuracies of 0.645 and 0.68, respectively, compared with only 0.005 and 0.045 for the untrained ViT-B. Appendix D.3 further shows that the matched cosine similarity remains almost unchanged when ImageNet images with detected text are excluded (0.234 to 0.23).
- The core method is clear. By subtracting a matched phase-scrambled control, the authors control reasonably well for background and rendering effects. The untrained ViT baseline, spelling clusters, and word-length-conditioned permutation tests also reduce the likelihood that the main results can be explained solely by visual similarity between word forms.
Critical weaknesses
- The evidence for “consistent vector offsets” is insufficient. Figure 1b shows only six hand-selected examples from the 60 country–capital pairs. Appendix C.1 further explains that the horizontal axis is defined using the mean offset of the undisplayed pairs and is visually stretched by a factor of 1.6. Meanwhile, Section 7 acknowledges that these offsets do not improve analogy retrieval. I understand that Figure 1b may primarily be intended as an illustration. Nevertheless, the current evidence is insufficient to support the claim in the Abstract and Section 1 that the model exhibits vector-offset relations comparable to those in classical word embeddings. The authors should quantify offset consistency over all pairs and compare it with random-pair or label-shuffled baselines. Otherwise, this claim should be weakened.
- “Learn word meaning” is broader than what the experiments directly establish. In Table 1, synonym top-10 accuracy is only 0.06 for DINOv3-H and 0.1 for DINOv3-7B. Appendix C.1 and Table 5 also show that, after correction, no vision model reliably ranks synonyms above antonyms. The current results provide strong evidence for categorical and factual associations and for word–visual alignment. However, they may still arise from word identification combined with scene-text co-occurrence in the training images, a possibility also acknowledged in Section 7. The Abstract, and Conclusion should therefore distinguish more clearly among lexical association, visual grounding, and broader word meaning.
Questions for authors
-
Can the authors quantify offset consistency over all 60 country–capital pairs? If these offsets do not improve analogy retrieval, what do they establish beyond ordinary country–capital association?
-
Were the vocabularies, relation pairs, and class names described in Appendices A and D.1 fully fixed before inspecting the results of any evaluated model?
Thank you for the careful review, 7ta7, and the kind words. We're glad you found the core method clear and the word-class alignment results convincing.
W1. Evidence for consistent vector offsets
We quantified offset consistency over all 60 country-capital pairs. For each pair, we take the cosine between its country-to-capital offset and the mean offset of held-out pairs (five folds), and average over pairs. As you suggested, we compare this with 999 random re-pairings of countries and capitals.
The offsets are clearly more consistent than under random re-pairing in both DINOv3 models, CLIP-B and Qwen3-8B, but not in the untrained ViT-B. The six pairs in Figure 1b therefore illustrate a regularity present across all pairs.
We will keep the offset finding, but the Abstract and Introduction will no longer compare the offsets to those of word embeddings trained on text. The full analysis will be added to Appendix C.
W2. "Learn word meaning" is broader than the experiments
The paper now makes this distinction explicit (see our response to Z4e4, W2). It defines word meaning as lexical meaning and tests two parts: how a word relates to other words, and whether it relates to images of the category it names. The Abstract now names these two parts more generally for readability, and the Conclusion details the distinction in our claims following your recommendations.
Synonymy is the weakest of the relations we test, as you note. Section 7 now gives an account of why: the models may learn which words occur together, and synonyms tend to replace each other rather than appear together. In line with this, DINOv3-H relates some synonyms of class names to the right photographs, such as "maize" for the class corn (see our response to e2jz, W1).
We also tested the role of text in the training images directly (see our response to reviewer Z4e4, W3). Of two Web-DINO-3B models trained with the same objective, the one trained only on images that contain text retrieves a class from its written name (word-to-class top-10 of 0.355), while the one trained on all images does not (0.010, against 0.005 for the untrained ViT-B). Text in the training images is therefore one source of the structure, as you suggest.
Our Spanish experiment (see our response to reviewer e2jz, W2) shows that the grounding does not depend on English spelling: in DINOv3-H, Spanish class names that share no letter pairs with their English names still retrieve their class (20 of 82 in the top ten, against 0.8 by chance), whereas category structure among such Spanish nouns is not detectable.
Q1. Offset consistency over all 60 country-capital pairs
Please see response in W1. We read the offsets as a geometric regularity of the country-capital association, not as analogical reasoning. This agrees with Section 7, which states that the offsets do not improve analogy retrieval.
Q2. Were the word lists fixed before any result was inspected?
Yes. The vocabulary and the relation pairs were fixed before any model was evaluated, and the class names were fixed by the rules in Appendix D.1 before any model was evaluated on them. None of these rules uses the output of an evaluated model. Two later choices are disclosed in the paper: the six pairs shown in Figure 1b and six two-clue pilot grids, which are excluded from the results. The lists added for this rebuttal were likewise fixed before rendering.
For the discussion period, we have updated the main text of the PDF (in blue), and we will add the new analyses to the appendices in full for the camera-ready version. We hope these changes address your questions and that you will consider raising your score.