- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper developed a family of unified image-video encoders and a training recipe using public data and priors from released models. The team conducted very comprehensive evaluations and ablations against the state of the art open-source encoders (some of them are open dataset). The results demonstrated the overall improvements compared with other models.
Critical strengths
- This paper made data curation and experiment setup very clear and detailed. This helps with the reproduction of the proposed methods and work pipeline
- The paper introduces reference tokens whose outputs are mapped through a trainable student projector. The distillation uses the reference tokens.
- The proposed snapshot-replay improves image and video performance over continuous video-only fine-tuning.
Critical weaknesses
- In benchmark tasks such as image classification and retrieval, video classification and retrieval, and multimodal understanding, the proposed framework does not show significant performance improvements against baselines
Questions for authors
- Even it's not a fair comparison, I would like to see how the proposed encoder benchmarks against some close-sourced commercial models.
2 comments
Thanks a lot for the detailed responses. I maintain my score at 3. This is a good paper.
Dear Reviewer e5py,
Thank you for your careful review. We appreciate your recognition of the clarity of the data curation and experimental setup, the reference-token supervision design, and the improvements from Snapshot-Replay over continuous video fine-tuning. We address improvement magnitude and commercial encoders below.
Response. Improvements vary across tasks and baselines, with clear gains in image retrieval and robustness benchmarks. Our goal is competitive performance with public direct data and released priors. Representative results follow (Tables 1, 2, and 3). Image/video metrics are percentages. Multimodal Overall averages nineteen tasks normalized to 0 to 100.
The gains span classification, retrieval, and multimodal understanding. COKE-H improves ImageNet-A by 5.1 points over SigLIP 2-SO, and all four retrieval directions improve. Video comparisons match spatial patch counts per encoding unit but use different frame counts. Multimodal comparisons share the LLaVA-NeXT/Qwen2.5-7B-Instruct protocol, with additional REG/REF outputs in COKE (Appendices C.2, C.3, and C.5).
Controlled experiments provide design evidence beyond comparisons between complete encoders. With teacher, distillation weight, and budget matched, the reference-token design gains 2.80/5.20 classification/retrieval points over pooled-patch distillation, including the effect of additional REF tokens (Table 4(b)). With image initialization and video budget matched, Snapshot-Replay gains 1.81/1.03 image/video points over continuous fine-tuning (Table 5(a)).
The complete recipe combines competitive performance with experimentally supported training designs. Some COKE-G video configurations remain below PE-G, and COKE-H's Square OCR/chart/document mean is below SigLIP 2-SO. The classification, retrieval, and multimodal gains, together with controlled ablations, support the value of our recipe.
Response. The submitted paper already includes the publicly reported Seed-ViT results in Appendix C.2, Table 10, as a commercial visual-encoder reference. Fully closed-source systems do not expose features for unified encoder evaluation or substitution into the same LLM. SigLIP 2 and PE have accessible weights despite nonpublic pretraining data and are evaluated at encoder level.
Seed-ViT provides a reference associated with a commercial multimodal system. It is the 532M-parameter encoder used by Seed1.5-VL. Its reported classification checkpoint precedes LLM integration (technical report Section 6.1.1, Table 5) [1]. The encoders have similar scales and support native inputs. COKE-H Native exceeds Seed-ViT on all six reported classification benchmarks. Scores below are percentages.
COKE-H Native's classification mean is 2.9 points higher than the reported Seed-ViT baseline. The baseline comes from the technical report and corresponds to the encoder before LLM integration.
We hope these comparisons clarify your questions. If they do, we would appreciate your consideration of updating your score. We welcome further discussion.
References