acceptodds
Under review as a conference paper at ICLR 2027

COKE: A Unified Image-Video Encoder Trained on Public Corpus

Abstract

Leading vision encoders often rely on training corpora that are not publicly released, limiting access to comparable training resources. We present **COKE**, a family of unified image-video encoders and a training recipe using public data and priors from released models. The family includes COKE-ViT-H (0.6B) and COKE-ViT-G (1.9B), with shared image-video weights at each resolution. ***Visual Reference Pretrain*** applies auxiliary visual supervision through dedicated reference-token outputs and improves early sample efficiency. At a fixed distillation weight, this pathway achieves higher classification and retrieval scores than pooled-patch distillation. ***Snapshot-Replay*** periodically interpolates video-trained weights with a fixed image-pretrained checkpoint and resumes optimization from the fused weights. In controlled COKE-ViT-H-256 experiments, it improves image and video averages over continuous video-only fine-tuning by 1.81 and 1.03 points, respectively, without replaying image-text data. A resolution curriculum supports fixed and native inputs, while COKE-K and COKE-V guide concept balancing alongside quality filtering and deduplication to construct the 1.68B-pair **COKE-1.7B** corpus. Fixed-budget ablations support COKE-K vocabulary selection and the utility of a data mixture incorporating COKE-V-selected samples. At 512 resolution, COKE-CLIP-G achieves 86.8 average zero-shot accuracy across six image classification benchmarks and 79.9 average retrieval Recall@1, compared with 86.6 and 78.9 for PE_core-G under our evaluation protocol. Frozen-encoder probing and multimodal evaluations further demonstrate the utility of the learned representations beyond zero-shot image-text tasks. We will release our models, vocabularies, and evaluation code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Foundation Models, Contrastive Vision-Language Learning, Unified Image-Video Encoders

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_e5pyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper developed a family of unified image-video encoders and a training recipe using public data and priors from released models. The team conducted very comprehensive evaluations and ablations against the state of the art open-source encoders (some of them are open dataset). The results demonstrated the overall improvements compared with other models.

Critical strengths
  • This paper made data curation and experiment setup very clear and detailed. This helps with the reproduction of the proposed methods and work pipeline
  • The paper introduces reference tokens whose outputs are mapped through a trainable student projector. The distillation uses the reference tokens.
  • The proposed snapshot-replay improves image and video performance over continuous video-only fine-tuning.
Critical weaknesses
  • In benchmark tasks such as image classification and retrieval, video classification and retrieval, and multimodal understanding, the proposed framework does not show significant performance improvements against baselines
Questions for authors
  • Even it's not a fair comparison, I would like to see how the proposed encoder benchmarks against some close-sourced commercial models.
2 comments

Dear Reviewer e5py,

Thank you for your careful review. We appreciate your recognition of the clarity of the data curation and experimental setup, the reference-token supervision design, and the improvements from Snapshot-Replay over continuous video fine-tuning. We address improvement magnitude and commercial encoders below.

Q1: Improvements across tasks and empirical conclusions.

Response. Improvements vary across tasks and baselines, with clear gains in image retrieval and robustness benchmarks. Our goal is competitive performance with public direct data and released priors. Representative results follow (Tables 1, 2, and 3). Image/video metrics are percentages. Multimodal Overall averages nineteen tasks normalized to 0 to 100.

ComparisonMetricBaselineCOKEDifference
H-384 vs. SigLIP 2-SO-384Image classification mean84.685.9+1.3
H-384 vs. SigLIP 2-SO-384Image retrieval mean77.480.4+3.0
H-384 vs. PE-L-336Video classification mean72.874.0+1.2
H-384 vs. PE-L-336Video retrieval mean54.756.0+1.3
G-512 vs. SigLIP 2-g-512, SquareOverall (19 tasks)57.6958.64+0.95

The gains span classification, retrieval, and multimodal understanding. COKE-H improves ImageNet-A by 5.1 points over SigLIP 2-SO, and all four retrieval directions improve. Video comparisons match spatial patch counts per encoding unit but use different frame counts. Multimodal comparisons share the LLaVA-NeXT/Qwen2.5-7B-Instruct protocol, with additional REG/REF outputs in COKE (Appendices C.2, C.3, and C.5).

Controlled experiments provide design evidence beyond comparisons between complete encoders. With teacher, distillation weight, and budget matched, the reference-token design gains 2.80/5.20 classification/retrieval points over pooled-patch distillation, including the effect of additional REF tokens (Table 4(b)). With image initialization and video budget matched, Snapshot-Replay gains 1.81/1.03 image/video points over continuous fine-tuning (Table 5(a)).

The complete recipe combines competitive performance with experimentally supported training designs. Some COKE-G video configurations remain below PE-G, and COKE-H's Square OCR/chart/document mean is below SigLIP 2-SO. The classification, retrieval, and multimodal gains, together with controlled ablations, support the value of our recipe.

Q2: Commercial visual-encoder comparisons.

Response. The submitted paper already includes the publicly reported Seed-ViT results in Appendix C.2, Table 10, as a commercial visual-encoder reference. Fully closed-source systems do not expose features for unified encoder evaluation or substitution into the same LLM. SigLIP 2 and PE have accessible weights despite nonpublic pretraining data and are evaluated at encoder level.

Seed-ViT provides a reference associated with a commercial multimodal system. It is the 532M-parameter encoder used by Seed1.5-VL. Its reported classification checkpoint precedes LLM integration (technical report Section 6.1.1, Table 5) [1]. The encoders have similar scales and support native inputs. COKE-H Native exceeds Seed-ViT on all six reported classification benchmarks. Scores below are percentages.

MetricSeed-ViT (reported)COKE-H NativeDifference
Six-benchmark classification mean82.585.4+2.9
ImageNet-A85.591.6+6.1
ObjectNet79.283.8+4.6

COKE-H Native's classification mean is 2.9 points higher than the reported Seed-ViT baseline. The baseline comes from the technical report and corresponds to the encoder before LLM integration.

We hope these comparisons clarify your questions. If they do, we would appreciate your consideration of updating your score. We welcome further discussion.

References

  • [1] Seed1.5-VL Technical Report.

Thanks a lot for the detailed responses. I maintain my score at 3. This is a good paper.

Reviewer_MdpFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents COKE, a family of unified image-video vision encoders trained on publicly available image-text and video-text datasets. The authors propose Visual Reference Pretrain (VRP) to leverage a frozen DINO teacher to improve pre-training efficiency and the quality of learned representation via knowledge distillation. They also introduce the Snapshot-Replay, which periodically interpolates video-finetuned parameters with a fixed image-pretrained checkpoint and resumes optimization from the interpolated weights, aiming to preserve image understanding capabilities during video adaptation. They construct a 1.7B dataset by collecting and processing several public datasets to train their vision encoder. Experimental results show that the proposed model exceeds existing vision encoders (SigLIP, UniViTAR, etc.) on zero-shot classification and retrieval tasks. And the proposed encoder can be applied to multimodal large language models like LLaVA-NeXT to perform visual understanding/reasoning/OCR tasks, surpassing other existing vision encoders as well. The paper is organized and written well.

Critical strengths
  1. A strong vision encoder trained from public datasets is valuable to the research community. So is the constructed dataset.
  2. The performance is good, outperforming several existing vision encoders.
Critical weaknesses
  1. The methodological novelty of the individual contributions is somewhat limited. Several proposed components are closely related to established techniques. VRP combines dedicated distillation tokens, pretrained visual teacher supervision, and contrastive learning. Snapshot-Replay is based on repeated parameter interpolation toward a fixed pretrained checkpoint, conceptually related to weight interpolation and pretrained-weight regularization. The data curation pipeline largely follows established vocabulary-based sampling and filtering strategies, with newly constructed concept vocabularies. While their integration and the resulting model is valuable, the paper does not sufficiently establish which conceptual advances distinguish these components from existing approaches beyond their particular configurations.
  2. In VRP, DINO teacher is introduced to provide vision knowledge during the pre-training of COKE. And DINO is trained using several other datasets (including internal, private ones) rather than the 1.7B dataset constructed in this work. Therefore, is it reasonable to claim that COKE is trained on public corpus? Meanwhile, from the experimental results in section 3.3.1, appendix D.2 and C.6.5, it seems that the strong performance of COKE is sourced directly from the DINO distillation. Then, what is the core technical contribution of this work? Knowledge distillation itself is useful. But for this paper, as claimed as vision encoder trained on public corpus, I expect that the whole training procedure is performed on the 1.7B dataset without any distillation.
  3. COKE is described as a unified image-video vision encoders. However, for video representation learning, COKE is not specifically optimized. The proposed Snapshot-Replay fine-tuning mechanism aims at adapting the image encoder to videos without extra temporal modules and keeping the image understanding capability during this adaptation, instead of improving the video semantic modeling. I expect more video-targeted modules or algorithms, maybe even for long videos.
Questions for authors

Please refer to weaknesses.

3 comments

Dear Reviewer MdpF,

Thank you for your careful review and for recognizing the encoder's performance and community value. Your concerns identify two distinctions that our presentation should make clearer: public direct training corpora versus pretrained model priors, and shared image-video encoding versus long-range temporal modeling. We address these first principles alongside the controlled evidence for our training designs below.

Q1: Public training corpora, released model priors, and the role of DINO.

Response. Our setting combines public direct training corpora with released model priors, whose external pretrained knowledge we explicitly acknowledge. We combine public direct image-text and video-text corpora with released auxiliary models, without accessing their pretraining corpora. DINOv3 supplies visual supervision and SigLIP 2 supplies text initialization and quality filtering, both carrying external pretrained knowledge. We will clarify in the abstract, introduction, and corpus-size comparisons that COKE-1.7B counts direct image-text pairs, not auxiliary pretraining data. These comparisons assess complete training recipes, including their direct data and released model priors (Appendix D.6).

Our technical focus is integrating teacher knowledge with image-text alignment. The same-teacher comparisons in Q2 show how supervision design affects classification and retrieval. In the main recipe, distillation ends after 8B of the 28.8B image-text samples, followed by 20.8B contrastive-only samples. The later stage continues optimizing the teacher-guided representation for image-text alignment without teacher computation (Section 2.2, Table 8).

Our controlled budget experiments demonstrate improved early sample efficiency with VRP. Both SO400M-384 runs match REG/REF tokens, initialization, and optimization (Appendix D.2). VRP reaches 76.4 mean classification accuracy at 0.6B samples and 82.6 at 1.6B, versus 75.5 without VRP at 1.6B. In the fixed-50M-pool experiment, VRP achieves classification/retrieval means of 65.8/69.9 at 200M processed samples, versus 57.9/67.0 without VRP. Longer no-VRP runs can exceed the short-budget VRP reference. Although these runs do not determine the teacher's final full-scale contribution, the matched-budget results show clear early gains from VRP. See Reviewer 3Wks, Q3 for the complete budget comparison.

Q2: Technical distinctions beyond established distillation, interpolation, and sampling.

Response. VRP improves early sample efficiency by combining visual-teacher guidance with image-text alignment from the start of training. Building on the distillation token design in DeiT (Appendix A.2), VRP applies frozen-teacher supervision to REF outputs while pooled patch outputs align with text. Both objectives jointly update the shared backbone through attention. The budget experiments in Q1 demonstrate faster early convergence in terms of processed samples, while the same-teacher comparisons below show that REF supervision achieves better classification and retrieval than pooled-patch distillation.

The same teacher can improve classification while harming retrieval, so the supervision design matters. Table 4(b) matches the teacher, distillation weight, and 1.6B-sample budget. Means are percentages. All rows use four REG tokens.

ConfigurationREF tokensClassificationRetrieval
No distillation076.4873.83
Pooled-patch distillation079.5569.85
Reference-token distillation882.3575.05

Pooled-patch distillation raises classification but lowers retrieval despite separate projection heads. REF avoids this loss and gains 2.80/5.20 classification/retrieval points over pooled-patch distillation. This tests the combined token-and-interface design, not output choice independently of the eight added REF tokens. In a separate 400M-sample weight sweep, lowering the pooled-patch weight does not close the gap. REF exceeds the best respective pooled-patch classification/retrieval means by 1.38/2.33 points within the tested range (Table 4(c), Appendix D.5).

Snapshot-Replay anchors video optimization to a fixed image checkpoint through periodic weight interpolation. WiSE-FT interpolates after fine-tuning, while Lookahead resumes from weights interpolated with an evolving slow state. Snapshot-Replay repeatedly blends with the fixed image checkpoint and resumes video training. This shares the anchoring motivation of pretrained-weight regularization, implemented through post-burst interpolation rather than L2-SP's loss penalty. Appendix D.3 gives a regularized-minimization interpretation of the fusion step. Our contribution is the fixed-checkpoint, resumed-training strategy, supported by matched comparisons with continuous fine-tuning, Lookahead, and WiSE-FT in Q3.

(Our response continues in Part 2 below.)

(Response continued from Part 1.)

Our data contribution is the construction and controlled evaluation of knowledge-based concept vocabularies. Matching and sampling follow MetaCLIP. Without VRP, replacing MetaCLIP's vocabulary with COKE-K raises the ten-metric image mean from 70.78 to 71.58, with the candidate pool, selected pool size, sampling cap, and 1B-sample budget matched (Table 6, Appendix C.6.9). The text tower and filtering use released priors, while this comparison isolates the COKE-K vocabulary replacement without VRP. COKE-V provides complementary visual categories and is evaluated with its smaller selected pool and repeated sampling. The controlled COKE-K comparison establishes a vocabulary-based data-selection benefit.

Q3: Unified image-video encoding and video-targeted optimization.

Response. Our video objective is to improve video performance while preserving image capability within a shared encoder. At each resolution, images and videos share the patchifier, backbone, and parameters. Two consecutive sampled frames map jointly to cube tokens. Images use the same path via frame duplication. This can encode within-pair information, but cubes have no cross-cube attention and their outputs are averaged, making the descriptor invariant to cube permutations. We do not claim long-range temporal-order modeling or demonstrated long-video capability (Section 2.1, Appendix B.1).

Video-text training and Snapshot-Replay improve video performance, not only image retention. Each video branch starts from its image checkpoint and trains for video-text alignment. Table 5(a) matches architecture, image initialization, video data, and the 640M-sample budget. Image/video means equally average ten/nine metrics (%).

ConfigurationImage meanVideo mean
Image-pretrained checkpoint81.5556.23
Continuous video fine-tuning81.2161.20
Lookahead81.5161.31
WiSE-FT (β = 0.5)82.7961.01
Snapshot-Replay83.0262.23

Continuous fine-tuning improves video performance but lowers the image mean. Snapshot-Replay gains 1.81/1.03 image/video points over it. Across the WiSE-FT coefficient sweep, the best image mean is 82.79 at β = 0.5 and the best video mean is 61.77 at β = 0.8. Snapshot-Replay exceeds these separate maxima by 0.23/0.46 points in this setting (Appendix D.4).

We will clarify these contributions and scope in the abstract and introduction: the supported contributions are a supervision design that improves early sample efficiency while supporting both classification and retrieval, a video adaptation strategy that improves both image retention and video performance, and vocabulary-based curation from public direct corpora using released priors. We hope this evidence and the more precise scope support reconsideration of your recommendation. Thank you again for the constructive discussion.

The authors' responses address my concern well. Now I'd love to raise my rating. Hopefully this work can benefit the community.

Reviewer_3WksOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds COKE which is a family of unified image-video encoders trained on public data with priors from released models. The main contributions are Visual Reference Pretrain (distills a frozen DINOv3 teacher into dedicated reference tokens), Snapshot-Replay (adapts image checkpoints to video by periodically interpolating back toward the image weights) and COKE-1.7B (a corpus curated from public datasets using knowledge-based vocabularies). The largest model matches or slightly outperforms PE-G on zero-shot image classification and retrieval but trails it on video.

Critical strengths

The paper shows that competitive encoders can be trained on public data with priors from released models. COKE-ViT-H performs comparably to or better than SigLIP 2-SO on zero-shot classification and retrieval, and COKE-ViT-G performs comparably to PE-G on images. This offers a viable alternative to recipes that depend on private corpora.

The paper has two simple techniques that are likely useful beyond this recipe. In their ablations, distilling the teacher into dedicated reference tokens improves both classification and retrieval, while distilling into pooled patch features trades one for the other. Snapshot-Replay improves both image and video performance during video-only fine-tuning and outperforms Lookahead and WiSE-FT.

Critical weaknesses

The ablations do not identify where the gains come from. Most design choices are validated with runs of COKE-ViT-H at 256 resolution, and there is no full-scale model without VRP or ablation of the resolution curriculum. Since the paper’s contributions are these techniques, a full-scale baseline using the same priors and data without them is important to show that they drive the final performance.

The paper does not check for benchmark leakage. It selects some hyperparameters on the reported benchmarks. And it builds COKE-V from ImageNet class names. Since the paper’s claim is parity with encoders trained on private data, any of these could inflate the reported results.

Questions for authors

Clarifications

How was the 8B-sample length for VRP chosen? The ablations stop at 1.6B samples, and the gap between the VRP and non-VRP runs in Figure 6 shrinks after 0.4B. Is there evidence that VRP still matters at full scale? Were the training corpora checked for overlap with the evaluation sets? Some hyperparameters were also selected on the reported benchmarks, so how do we know the reported performance is not inflated? Why were DINOv3 and SigLIP 2 chosen as the teacher, text tower, and filtering model? What role do the REF tokens play after VRP, when they no longer have an objective? In the LLaVA experiments, how much of the gain comes from the 12 additional REG/REF tokens passed to the language model? Will the IDs or URLs of the selected image-text pairs and the training code be released?

Suggestions

A full-scale model without VRP or extend the Figure 6 comparison closer to 8B samples. Compare the joint training against distilling either by distilling first and then running contrastive training or by starting from DINOv3 weights. An ablation of the resolution curriculum. Dense-prediction evaluations such as segmentation and depth to better test COKE for a vision encoder for MLLMs. Results across more model sizes especially since COKE-ViT-G is worse than COKE-ViT-H at image retrieval.

4 comments

Dear Reviewer 3Wks,

Thank you for your detailed review and concrete suggestions. We appreciate your recognition of the feasibility of training with public data and released model priors, as well as the practical value of reference-token distillation and Snapshot-Replay. We address your concerns with overlap checks and cleaned-subset reevaluation, VRP budget comparisons, a LLaVA REG/REF output ablation, and ADE20k linear segmentation.

Q1: Component contributions and the complete training recipe.

Response. The controlled experiments evaluate individual designs, while the full-model experiments evaluate the complete recipe. Ablations use COKE-ViT-H (about 0.6B parameters) at 256 resolution (Appendix C.6).

  • VRP design. With the same teacher, distillation weight, and 1.6B-sample budget, reference-token distillation improves classification and retrieval means over pooled-patch distillation by 2.80 and 5.20 percentage points (Table 4(b)). This includes eight REF tokens and the supervision interface. A separate matched-token comparison supports early sample efficiency (Appendix D.2).
  • Snapshot-Replay. With matched image initialization and 640M video samples, it improves image and video means over continuous video fine-tuning by 1.81 and 1.03 percentage points (Table 5(a), Appendix C.6.7).
  • COKE-K. Without VRP, with candidate pool, selected pool size, sampling cap, and 1B-sample budget fixed, it improves the aggregate mean over MetaCLIP by 0.80 percentage points (Table 6, Appendix C.6.9).

Full H/G training costs about 10K/13K H800 device-days. Full-scale component-removal retraining has not been performed (Appendix C.1.2). See Q3 and Q9.

Q2: Evaluation overlap, category priors, and hyperparameter selection.

Response. The additional overlap checks and unified reevaluation on cleaned subsets preserve COKE-H's average classification and retrieval advantages.

The new overlap check covers eight image evaluation datasets. 64-bit pHash and Hamming distance screen candidates. DeepSeek-v4.1-flash distinguishes same-image variants from similar content. Counts are evaluation images overlapping COKE-1.7B. Rates use dataset totals.

Classification datasetOverlapTotal imagesRate
ImageNet-1K1,00050,0002.00%
ImageNet-A567,5000.75%
ImageNet-R1,94030,0006.47%
ImageNet-V224310,0002.43%
ImageNet-Sketch5,60850,88911.02%
ObjectNet050,2730.00%
Retrieval datasetOverlapTotal imagesRate
COCO Captions1545,0003.08%
Flickr30K261,0002.60%

The average advantages remain after removing the confirmed overlapping evaluation images. All models use unchanged checkpoints and identical cleaned subsets. Classification accuracy/retrieval Recall@1 are percentages. AVG-I-C/AVG-I-R equally average six benchmarks/four directions.

ClassificationSigLIP 2-SO-384PE-Core-L-336COKE-H-384
ImageNet-1K84.2583.5584.91
ImageNet-A87.0689.1792.25
ImageNet-R95.9695.0996.61
ImageNet-V278.6378.1279.96
ImageNet-Sketch76.0373.1975.84
ObjectNet85.4184.4785.99
RetrievalSigLIP 2-SO-384PE-Core-L-336COKE-H-384
Flickr30K T→I84.1185.4887.21
Flickr30K I→T94.8796.6196.82
COCO T→I56.9757.1759.68
COCO I→T73.6976.0678.21
Cleaned-subset meanSigLIP 2-SO-384PE-Core-L-336COKE-H-384
AVG-I-C84.5683.9385.93
AVG-I-R77.4178.8380.48

The observed average gains cannot be explained simply by the overlaps detected in this check. Classification/retrieval gains are 1.37/3.07 points over SigLIP 2 and 2.00/1.65 over PE-Core-L. They persist on high-overlap ImageNet-R, low-overlap ImageNet-A, and ObjectNet with no detected overlap. ImageNet-Sketch remains below SigLIP 2.

Public category vocabularies provide concept priors rather than supervision from test images. COKE-V matches public captions to ImageNet-21K names, without test images (Section 2.4, Appendix B.3). DataComp permits category labels, not test images [1]. Other tasks provide complementary evidence beyond ImageNet-category priors.

Hyperparameter selection will be disclosed separately from overlap checking. REG/REF counts and Snapshot-Replay parameters were chosen on ablation benchmarks without independent validation. K = 10 and α = 0.75 favor images, not the best video mean (Table 5(b,c)). Overlap removal does not resolve selection bias. Multimodal benchmarks were not used for selection. We will disclose both procedures and results.

(Our response continues in Part 2 below.)

(Response continued from Part 1.)

Q3: VRP duration and training-budget effects.

Response. The 8B-sample duration is an empirical switching point based on the main training dynamics. As training approached five epochs (about 8B samples), teacher gains plateaued and supervision slowed contrastive-loss convergence, prompting removal (Section 2.2).

The narrowing gap in Figure 6 is consistent with early guidance and improved sample efficiency. With visual initialization, text tower, tokens, and optimization matched, the SO400M-384 classification gap shrinks from 14.3 points at 0.4B to 7.1 at 1.6B. VRP reaches 76.4 at 0.6B and 82.6 at 1.6B, versus 75.5 without VRP at 1.6B. Teacher supervision remains throughout (Appendix D.2).

The additional experiment shows that a model without VRP can catch up to a short-budget VRP model with more training. All runs use a fixed 50M-pair subset randomly sampled from COKE-1.7B, with matched visual architecture, resolution, REG/REF tokens, and text tower settings. Visual backbones are randomly initialized, and each run decays its learning rate from 10⁻³ to zero. Budgets count processed samples, not distinct pairs.

ConfigurationEpochsSamplesClassificationRetrieval
With VRP4200M65.869.9
Without VRP4200M57.967.0
Without VRP8400M64.370.5
Without VRP16800M68.370.8

These results demonstrate VRP's early sample efficiency. At 200M samples, VRP improves classification/retrieval by 7.9/2.9 points. Without VRP, retrieval exceeds the 200M VRP reference at 400M, while classification remains 1.5 points lower. Both exceed this reference at 800M. This comparison uses a fixed 50M pair pool rather than full scale training. Together with Figure 6, these results show that VRP achieves stronger performance with fewer processed samples. We will include the setup and results in Appendix D.2.

Q4: Auxiliary model selection.

Response. The auxiliary models serve distinct roles and use released weights.

  • Visual teacher. DINOv3 provides self-supervised visual priors complementary to language alignment. With student backbone, REG/REF configuration, and budget matched, replacing DINOv2-g with DINOv3-7B raises the ten-metric image mean from 77.65 to 79.43, improving classification and retrieval. Projector output dimensions match teacher features (Appendix C.6.5, Table 17).
  • Text tower. SigLIP 2-SO400M-384 provides a pretrained text space for the randomly initialized visual backbone. With a trainable text tower, pretrained rather than random initialization raises the aggregate mean from 72.52 to 74.44. Freezing gives 74.73, with better classification but lower retrieval than updating it. We freeze it during VRP and update it later (Appendix C.6.6, Table 18, Section 2.2).
  • Quality filtering. The released siglip2-so400m-patch16-256 scores image-text similarity. We retain the top 40%, then deduplicate and balance by vocabulary (Section 2.4, Appendix B.3). Its image-text matching ability motivates this choice.

Q5: REF tokens after VRP.

Response. REF tokens continue to be optimized indirectly by the image-text contrastive objective after direct distillation ends. We remove the teacher and distillation loss while retaining all eight REF tokens. REF and REG tokens participate in attention but are excluded from patch pooling, receiving contrastive gradients through interactions with patch tokens.

Retaining REF tokens preserves a teacher-guided training state rather than fixed DINOv3 features. They adapt without reinitialization. REF/REG have similar computation but different training histories (Section 2.2, Appendix B.1).

Q6: REG/REF outputs in LLaVA.

Response. Omitting REG/REF outputs from the language model reduces Overall by 0.25 points, while COKE-H retains an advantage over SigLIP 2-SO. Two LLaVA models are separately trained and evaluated with or without REG/REF outputs, using the same COKE-H-384 checkpoint and LLaVA-NeXT/Qwen2.5-7B-Instruct protocol. Both retain four REG and eight REF tokens internally. Excluding their outputs leaves 576 patch tokens for the language model, matching the Square baselines' input token count. Scores are normalized to 0 to 100. Overall equally averages nineteen tasks (Table 3, Appendix C.5).

MetricWith outputsWithout outputsDifference
Overall (19 tasks)55.1954.94+0.25
General understanding mean60.3860.16+0.22
OCR/chart/document mean53.8053.44+0.36
Reasoning mean47.2547.10+0.15

The additional outputs contribute to the average gain but do not account for all of it. SigLIP 2-SO scores 54.66. COKE's gap falls from 0.53 to 0.28, a reduction of about 47%. Outputs add 1.4 on MMBench but subtract 0.6 on POPE.

(Our response continues in Part 3 below.)

(Response continued from Part 2)

Q7: Code and sample-list release.

Response. We will release training/evaluation code, weights, and COKE-K/COKE-V vocabularies, but not selected sample IDs or URLs. Source datasets, auxiliary versions, and preprocessing, filtering, matching, sampling, and deduplication settings support reproduction (Appendices B.2, B.3, C.1).

Q8: Joint training and alternative teacher-transfer strategies.

Response. Joint training allows visual prior transfer and language alignment to proceed together from the beginning. The teacher guides early learning until VRP ends (Section 2.2). Sequential distillation and DINOv3 initialization remain untested.

Feature distillation preserves independence between the student architecture and teacher weights. Direct initialization requires compatible structures and shapes, or weight adaptation. These alternatives merit controlled comparison.

Q9: Resolution curriculum.

Response. The resolution curriculum follows an established strategy for computational efficiency and input adaptation. We have not performed an independent curriculum ablation. Training progresses from 256 to 384, 512, and native resolution, allocating most samples to lower resolution before adapting to finer detail and variable aspect ratios. Figure 3 and Table 8 document the stages and sample allocation.

Q10: Dense representation transfer through segmentation.

Response. COKE-G achieves 47.1 mIoU on ADE20k with linear probing on frozen patch features, exceeding reported SigLIP 2-g/PE-Core-G scores by 4.4/8.2 points. We follow the DINOv3 protocol (Section 6.1.2, Appendix D.1) [2]. Baselines are reported in DINOv3 Table 3, not reevaluated.

ModelScale/patchHeadTesting or sourcemIoU (%) ↑
SigLIP 2g/16LinearDINOv3 Table 342.7
PE-CoreG/14LinearDINOv3 Table 338.9
COKEG/16LinearStandard47.1
COKEG/16LinearMulti-scale (+ms)50.8
DINOv37B/16LinearDINOv3 Table 355.9

The linear-head results support transferring COKE's spatial features to pixel-level semantic prediction. DINOv3-7B provides a larger-scale self-supervised reference at 55.9 mIoU. COKE-G reaches 47.1 with standard testing and 50.8 with multi-scale testing, a 3.7-point gain using the same head. Together with the reported SigLIP 2 and PE-Core results, these scores demonstrate the utility of COKE's patch features beyond global image-text tasks.

The segmentation results extend our transfer evaluation from nineteen LLaVA-NeXT tasks to dense prediction. Depth estimation was not evaluated. We will add the segmentation protocol and results to the appendix.

Q11: More model sizes and H/G image retrieval performance.

Response. H/G results show different classification and retrieval trends. Across four input settings in Table 1, G improves classification but trails H by 0.3 to 0.5 points in mean retrieval. In Table 18, freezing versus updating the pretrained text tower raises classification by approximately 1.28 points but lowers retrieval by 1.20. H and G share text architecture and initialization. We hypothesize that text configuration and joint adaptation may influence the observed scaling trends, motivating further investigation of joint visual and text scaling.

We will add these experiments. If they address your concerns, we would appreciate your consideration of updating your score. We welcome further discussion.

References

  • [1] DataComp: In Search of the Next Generation of Multimodal Datasets.
  • [2] DINOv3.

Thank you for the detailed response and the additional experiments. This is a good contribution, and I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.