acceptodds
Under review as a conference paper at ICLR 2027

LexDINO: Language-Grounded Prototypes for Visual Pretraining

Abstract

A general-purpose visual encoder should transfer effectively across recognition, dense prediction, and generative vision-language tasks. Existing pretraining approaches offer complementary strengths across these settings: visual self-supervision (e.g., DINO) is particularly effective for dense prediction, while image-text supervision (e.g., CLIP) often yields better transfer to generative vision-language models (VLMs). We introduce LexDINO, which brings language-derived semantic structure into DINO's visual self-distillation to improve vision-language transfer while retaining its superior visual capabilities. An independent caption corpus is clustered in an LLM's feature space, and the resulting centroids replace DINO's learned image-level prototypes. A second bank represents the same caption groups in a frozen image-text model's joint space, allowing its image tower to ground the corresponding DINO assignments. Semantic guidance therefore acts through DINO's own prediction space, alongside cross-view self-distillation and masked patch learning, without directly regressing backbone features or requiring captions paired with the training images. With the ImageNet-1k image corpus, backbone, and training schedule held fixed, LexDINO improves over a DINOv3 trained in the same way by 2.0 points in k-NN accuracy and 3.3 points in ADE20K mIoU. Under the same downstream VLM training recipe, it improves all twelve benchmarks, including a 25.4-point gain in the five-task OCR average, and its LLM-derived prototype bank outperforms a SigLIP2-text bank by 6.7 OCR points. At scale, a roughly 1B parameter LexDINO encoder, trained with a fraction of DINOv3's pretraining compute, achieves best or tied performance on all five dense benchmarks among the evaluated models below 2B parameters. Batch-independent sparse targets and full-grid patch supervision simplify its distillation into ViT-L/16 and ViT-B/16 students. Both students outperform size-matched DINOv3 encoders in recognition and VLM transfer while retaining strong dense prediction.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, visiona language models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KEE1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LexDINO modifies DINO-style visual self-distillation by replacing the learned image-level prototype bank with fixed centroids derived from clusters of an independent caption corpus in an LLM feature space. To enable learning a structure, for which a single target prototype does not suffice, top-k routing targets supervise LexDINO. The results show gains over a DINOv3 baseline, particularly for OCR-heavy VLM tasks.

Critical strengths

The paper studies an interesting tension: DINO-style encoders are strong for dense visual tasks, whereas image-text encoders are often better VLMs and provide better text-image alignment. The proposed shared group index is a good concept and addresses the ambiguity that a fixed semantic bank would leave unresolved.

The experiments cover a wide spectrum: recognition, segmentation, depth, general VQA/reasoning, OCR/document tasks, multiple backbone sizes, and multiple downstream language models.

The authors examine various interesting elements: feature distillation versus index alignment, prototype representation, caption source.

The results are convincing.

Critical weaknesses

The paper contrasts against DINOv3, a projected SigLIP2 feature loss, and several released encoders. It does not compare against dino.txt-style DINO-language alignment, EVA alternatives, or a paired/unpaired contrastive alternative under comparable data and compute. Such comparisons would provide the argument that the shared-index formulation is beneficial rather than one effective way of injecting language.

If grouping using language is the way to go, could be inferred from non-language sources to guide the self-distillation, e.g. SigLIP2-defined grouping or another alignment without injecting LLM information.

I would be very interested in the resulting prototypes: are they interpretable, visually grounded, balanced, (non-)redundant?

Questions for authors

How much of the controlled OCR gain is attributable to the frozen SigLIP2 router rather than to the LLM-derived prototype geometry?

What happens if caption groups are formed directly in SigLIP2 text space, using the same K and cluster-size controls, instead of retaining Qwen assignments?

Can you report downstream results for a random fixed prototype bank and for the alignment term with random or permuted shared indices? This would isolate geometry, stable identities, and router supervision.

Can you diagnose the OCR gains?

1 comment
Reviewer_KEE1OpenReview ↗

Reply to Rebuttal

I am happy with the rebuttal. It isolates the contributions of the paper more precisely, by comparing to alternatives, and cross-checking multiple technical options. The gains are now smaller compared to those alternatives, but now they can be attributed to the technical novelties in the paper, and overall it yields stable improvements and is therefore more convincing. I have also read the other reviews. Altogether it made me conclude to increase my score to a strong accept.

Reviewer_tUNVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LexDINO, a visual pretraining framework that incorporates language-derived semantic prototypes into DINO-style self-supervised learning. It replaces learnable image-level prototypes with fixed LLM-derived centroids and uses a frozen SigLIP2 encoder for cross-space alignment. Experiments demonstrate improvements over DINOv3 in recognition, dense prediction, and downstream VLM tasks, particularly OCR-related benchmarks.

Critical strengths
  • Interesting language-grounded prototype learning. Integrating LLM-derived semantic prototypes into DINO's self-distillation objective provides an interesting alternative to direct feature distillation.

  • Strong empirical performance. LexDINO improves recognition, dense prediction, and VLM transfer, with particularly substantial gains on OCR benchmarks. The controlled experiments and large-scale results demonstrate its potential as a general-purpose visual encoder.

Critical weaknesses
  • Insufficient explanation of improvements in dense prediction. Language-derived prototypes supervise image-level CLS features, while the patch-level iBOT objective remains largely unchanged. It is unclear why such global semantic supervision substantially improves segmentation and depth estimation. More targeted analyses are needed to explain these gains.

  • Insufficient isolation of language priors and limited recognition evaluation. The improvements may partly result from using fixed pretrained semantic prototypes and additional SigLIP2 supervision, rather than the specific benefits of LLM-derived representations. The reported training efficiency is also difficult to attribute to the proposed language priors. Furthermore, recognition evaluation is largely limited to ImageNet-1k, and the large-scale training data composition is not sufficiently detailed.

Questions for authors

Q1. Why do language priors improve dense prediction? Language-derived semantic priors would intuitively be expected to benefit high-level semantic recognition rather than fine-grained spatial perception. However, the paper reports substantial improvements in segmentation and depth estimation. Could the authors explain why language-grounded prototypes lead to these gains and provide further analysis or ablations to support this claim?

Q2. Limited classification evaluation. Given that language-derived prototypes primarily introduce semantic information, why is recognition evaluation limited to ImageNet-1k linear probing and k-NN? Could the authors include additional classification and fine-grained recognition benchmarks to validate generalization? Please also clarify the evaluation protocols and ensure consistent settings across compared models.

Q3. Are the Gains Attributable to Language-Derived or Pretrained Fixed Prototypes?.Are the improvements and efficiency gains due to LLM-derived semantics or simply using pretrained fixed prototypes? Could the authors compare against training a ViT-g from scratch with frozen DINOv3-7B prototypes under matched settings? Additionally, Table 12 retains Qwen-derived clusters for the SigLIP2 variant. Could the authors construct prototypes entirely from SigLIP2 text embeddings, including clustering, to isolate the contribution of LLM semantics?

Q4. Training data composition and reproducibility. Section 3.5 lists the training data sources but does not provide their exact composition. Could the authors report the sample counts, sampling ratios, and filtering procedures for each dataset in the approximately 320M-image mixture? A more detailed breakdown of the 8M-caption corpus used to construct language prototypes would also improve reproducibility.

Reviewer_ekpFOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Problem: The paper starts from a trade-off in visual encoders. Models trained from images alone can be strong at recognizing objects and their locations, yet less useful when paired with a language model for reading text, answering questions, or understanding charts and documents. Training on image-caption pairs can help with these language tasks, but may weaken spatial visual skills and requires paired captions.

Authors' idea: The authors ask whether a visual encoder can gain language-relevant concepts while keeping its recognition and spatial abilities, without needing a caption paired with every training image.

How LexDINO works: The authors use a separate collection of captions and a language model to group similar captions into concepts. A pre-trained image-and-text model then estimates which concepts fit each training image, and its scores guide LexDINO's predictions. LexDINO still learns from images by comparing different views and reconstructing hidden image regions; the training images do not need their own captions.

Why it matters: If successful, this approach could make one visual encoder useful both for tasks such as recognition and segmentation and as the image-reading component of a language model. The goal is to improve text and document understanding without giving up strong visual features.

Evaluation Setup: The authors evaluate the visual encoder on recognition and spatial tasks, then use it inside the same vision-language model (VLM) recipe to test language-related abilities:

  • Object recognition and spatial understanding: ImageNet classification is measured with a simple classifier and a nearest-neighbor test. ADE20K measures how accurately the model segments a scene into regions. In the controlled comparison with DINOv3, LexDINO improves by 2.2 and 2.0 percentage points on the ImageNet tests and by 3.3 points on ADE20K segmentation.
  • Vision-language tasks: Twelve tests cover visual question answering, reasoning, reading text in images, interpreting charts, and understanding documents. All twelve improve over the DINOv3 encoder using the same VLM training recipe; the average OCR score rises from 21.3 to 46.7.
Critical strengths
  • The approach could provide a versatile visual encoder for recognition and segmentation while also serving as the image-processing component of a language model.
  • The evaluation spans recognition, dense prediction, and vision-language tasks, with a consistent downstream VLM setup for the controlled comparison.
Critical weaknesses

A clearer, consolidated discussion of the method's limitations and trade-offs would help readers assess its practical costs, dependencies, and scalability.

Questions for authors
  1. What additional training costs and model dependencies does LexDINO introduce, including the compute required to create and use its language-derived concepts?
  2. How does LexDINO perform at scales larger than 2B parameters, and can those comparisons be controlled for training data and compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.